DistributeDefinitions is evaluating the definitions, and this only for a large number of definitions

I am using Mathematica 11.3 and this seems to me to be a bug. I would like, if possible, some idea on a workaround.

Here is an example of a trivial code that works as expected:

nI = 10;

(NM[#] :=  Print[#] ) & /@ Range[1, nI];

LaunchKernels;

DistributeDefinitions[NM];

That is, the code above generates no output, as expected.

Now, if the first line is changed to

nI = 20;

The same code leads to 40 lines being printed! From 1 to 20 two times.

For some reason, the DistributeDefinitions is forcing the definition of NM to be executed, and I do not want that to happen before I use ParallelSubmit and WaitAll. I tried this in two computers with Mathematica 11.3, any ideas on what is happening?

edited 2 days ago

asked Dec 28 at 20:00

Davi Rodrigues

1578

1

On my machine the behaviour changes from nI=17 to 18. If I look at Definitions[NM], up to 17 it prints as :=-definitions, but above that it prints as =-definitions. Thus the core issue may be not with the parallel tools, but with how definitions are stored.
– Szabolcs
2 days ago

1

The reason appears to be that Language`ExtendedFullDefinition switches from using RuleDelayed to Rule after 18 down-values. This causes DistributeDefinitions to subsequently leak the evaluation.
– Lukas Lang
2 days ago

Can you please report this to Wolfram support?
– Szabolcs
5 hours ago

add a comment |

I am using Mathematica 11.3 and this seems to me to be a bug. I would like, if possible, some idea on a workaround.

Here is an example of a trivial code that works as expected:

nI = 10;

(NM[#] :=  Print[#] ) & /@ Range[1, nI];

LaunchKernels;

DistributeDefinitions[NM];

That is, the code above generates no output, as expected.

Now, if the first line is changed to

nI = 20;

The same code leads to 40 lines being printed! From 1 to 20 two times.

edited 2 days ago

asked Dec 28 at 20:00

Davi Rodrigues

1578

1

On my machine the behaviour changes from nI=17 to 18. If I look at Definitions[NM], up to 17 it prints as :=-definitions, but above that it prints as =-definitions. Thus the core issue may be not with the parallel tools, but with how definitions are stored.
– Szabolcs
2 days ago

1

The reason appears to be that Language`ExtendedFullDefinition switches from using RuleDelayed to Rule after 18 down-values. This causes DistributeDefinitions to subsequently leak the evaluation.
– Lukas Lang
2 days ago

Can you please report this to Wolfram support?
– Szabolcs
5 hours ago

add a comment |

I am using Mathematica 11.3 and this seems to me to be a bug. I would like, if possible, some idea on a workaround.

Here is an example of a trivial code that works as expected:

nI = 10;

(NM[#] :=  Print[#] ) & /@ Range[1, nI];

LaunchKernels;

DistributeDefinitions[NM];

That is, the code above generates no output, as expected.

Now, if the first line is changed to

nI = 20;

The same code leads to 40 lines being printed! From 1 to 20 two times.

edited 2 days ago

asked Dec 28 at 20:00

Davi Rodrigues

1578

I am using Mathematica 11.3 and this seems to me to be a bug. I would like, if possible, some idea on a workaround.

Here is an example of a trivial code that works as expected:

nI = 10;

(NM[#] :=  Print[#] ) & /@ Range[1, nI];

LaunchKernels;

DistributeDefinitions[NM];

That is, the code above generates no output, as expected.

Now, if the first line is changed to

nI = 20;

The same code leads to 40 lines being printed! From 1 to 20 two times.

parallelization internals

edited 2 days ago

asked Dec 28 at 20:00

Davi Rodrigues

1578

edited 2 days ago

asked Dec 28 at 20:00

Davi Rodrigues

1578

edited 2 days ago

asked Dec 28 at 20:00

Davi Rodrigues

1578

asked Dec 28 at 20:00

Davi Rodrigues

1578

asked Dec 28 at 20:00

Davi Rodrigues

1578

1

On my machine the behaviour changes from nI=17 to 18. If I look at Definitions[NM], up to 17 it prints as :=-definitions, but above that it prints as =-definitions. Thus the core issue may be not with the parallel tools, but with how definitions are stored.
– Szabolcs
2 days ago

1

The reason appears to be that Language`ExtendedFullDefinition switches from using RuleDelayed to Rule after 18 down-values. This causes DistributeDefinitions to subsequently leak the evaluation.
– Lukas Lang
2 days ago

Can you please report this to Wolfram support?
– Szabolcs
5 hours ago

add a comment |

1

On my machine the behaviour changes from nI=17 to 18. If I look at Definitions[NM], up to 17 it prints as :=-definitions, but above that it prints as =-definitions. Thus the core issue may be not with the parallel tools, but with how definitions are stored.
– Szabolcs
2 days ago

1

The reason appears to be that Language`ExtendedFullDefinition switches from using RuleDelayed to Rule after 18 down-values. This causes DistributeDefinitions to subsequently leak the evaluation.
– Lukas Lang
2 days ago

Can you please report this to Wolfram support?
– Szabolcs
5 hours ago

On my machine the behaviour changes from nI=17 to 18. If I look at Definitions[NM], up to 17 it prints as :=-definitions, but above that it prints as =-definitions. Thus the core issue may be not with the parallel tools, but with how definitions are stored.
– Szabolcs
2 days ago

The reason appears to be that Language`ExtendedFullDefinition switches from using RuleDelayed to Rule after 18 down-values. This causes DistributeDefinitions to subsequently leak the evaluation.
– Lukas Lang
2 days ago

Can you please report this to Wolfram support?
– Szabolcs
5 hours ago

add a comment |

1 Answer
1

active

oldest

votes

TL;DR;

Execute the following code to fix the problem:

DistributeDefinitions;

DownValues[Parallel`Protected`DistDefs] =

  DownValues[Parallel`Protected`DistDefs] /.

   HoldPattern[

     u : Parallel`Parallel`Private`updates =

      rhs_Language`ExtendedFullDefinition

     ] :>

    (

     u = Replace[

       rhs,

       defs : {(_HoldPattern -> _) ..} :> With[

         {res = RuleDelayed @@@ Unevaluated@defs},

         res /; True

         ],

       {4}

       ]

     );

What does this do?

As mentioned in the comments, the issue is that Language`ExtendedFullDefinition seems to change the return format at 18 down-values. This causes the subsequent manipulations of DistributeDefinitions on the returned Language`DefinitionList[…] expression to leak the evaluation of the definitions (as their r.h.s. are no longer protected by the HoldRest attribute of RuleDelayed).

The code above fixes this by wrapping the call to Language`ExtendedFullDefinition (which happens in Parallel`Protected`DistDefs) with a piece of code that replaces definitions of the form HoldPattern[…]->… with HoldPattern[…]:>…, which prevents the evaluation leak.

answered 2 days ago

Lukas Lang

6,0531928

It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it?
– Davi Rodrigues
2 days ago

Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know
– Lukas Lang
2 days ago

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\$","\$"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "387"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f188517%2fdistributedefinitions-is-evaluating-the-definitions-and-this-only-for-a-large-n%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

TL;DR;

Execute the following code to fix the problem:

DistributeDefinitions;

DownValues[Parallel`Protected`DistDefs] =

  DownValues[Parallel`Protected`DistDefs] /.

   HoldPattern[

     u : Parallel`Parallel`Private`updates =

      rhs_Language`ExtendedFullDefinition

     ] :>

    (

     u = Replace[

       rhs,

       defs : {(_HoldPattern -> _) ..} :> With[

         {res = RuleDelayed @@@ Unevaluated@defs},

         res /; True

         ],

       {4}

       ]

     );

What does this do?

answered 2 days ago

Lukas Lang

6,0531928

It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it?
– Davi Rodrigues
2 days ago

Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know
– Lukas Lang
2 days ago

add a comment |

TL;DR;

Execute the following code to fix the problem:

DistributeDefinitions;

DownValues[Parallel`Protected`DistDefs] =

  DownValues[Parallel`Protected`DistDefs] /.

   HoldPattern[

     u : Parallel`Parallel`Private`updates =

      rhs_Language`ExtendedFullDefinition

     ] :>

    (

     u = Replace[

       rhs,

       defs : {(_HoldPattern -> _) ..} :> With[

         {res = RuleDelayed @@@ Unevaluated@defs},

         res /; True

         ],

       {4}

       ]

     );

What does this do?

answered 2 days ago

Lukas Lang

6,0531928

It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it?
– Davi Rodrigues
2 days ago

Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know
– Lukas Lang
2 days ago

add a comment |

TL;DR;

Execute the following code to fix the problem:

DistributeDefinitions;

DownValues[Parallel`Protected`DistDefs] =

  DownValues[Parallel`Protected`DistDefs] /.

   HoldPattern[

     u : Parallel`Parallel`Private`updates =

      rhs_Language`ExtendedFullDefinition

     ] :>

    (

     u = Replace[

       rhs,

       defs : {(_HoldPattern -> _) ..} :> With[

         {res = RuleDelayed @@@ Unevaluated@defs},

         res /; True

         ],

       {4}

       ]

     );

What does this do?

answered 2 days ago

Lukas Lang

6,0531928

TL;DR;

Execute the following code to fix the problem:

DistributeDefinitions;

DownValues[Parallel`Protected`DistDefs] =

  DownValues[Parallel`Protected`DistDefs] /.

   HoldPattern[

     u : Parallel`Parallel`Private`updates =

      rhs_Language`ExtendedFullDefinition

     ] :>

    (

     u = Replace[

       rhs,

       defs : {(_HoldPattern -> _) ..} :> With[

         {res = RuleDelayed @@@ Unevaluated@defs},

         res /; True

         ],

       {4}

       ]

     );

What does this do?

answered 2 days ago

Lukas Lang

6,0531928

answered 2 days ago

Lukas Lang

6,0531928

answered 2 days ago

Lukas Lang

6,0531928

answered 2 days ago

Lukas Lang

6,0531928

It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it?
– Davi Rodrigues
2 days ago

Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know
– Lukas Lang
2 days ago

add a comment |

It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it?
– Davi Rodrigues
2 days ago

Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know
– Lukas Lang
2 days ago

It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it?
– Davi Rodrigues
2 days ago

Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know
– Lukas Lang
2 days ago

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Mathematica Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Argthtjtr