Trying to make a “global embedding layer provider” in PyTorch
I am diving into a deep learning model. That model contains multiple classifier with same word embedding layer and can be trained simultaneously in different process. The original implementation from author uses distinct process, which just launch different training model and train it in a separate process. It is too memory consuming because each process/classifier use a embedding layer which requires load glove file larger than 4GB. I wonder if there is a way to make those process share their embedding layer so that the memory usage will be reduced.
I have used the multiprocessing lib but it didn't work. My solution is to put original training function into multiprocessing.Pool. For sharing embedding layer:
First, I use multiprocessing.Manager.dict(), but it throws Bad message length Error.
Then I tried to pass embedding layer as a parameter to the training function(It is passed by reference in default). It works well with a sample of glove file(10000 lines) but if I use full glove file(glove.42B.300d.txt), the process exits with 0 without error message, not entering training function.
I also tried use a global variable but the behavior is exactly like pass-to-function.
python python-multiprocessing pytorch
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
add a comment |
I am diving into a deep learning model. That model contains multiple classifier with same word embedding layer and can be trained simultaneously in different process. The original implementation from author uses distinct process, which just launch different training model and train it in a separate process. It is too memory consuming because each process/classifier use a embedding layer which requires load glove file larger than 4GB. I wonder if there is a way to make those process share their embedding layer so that the memory usage will be reduced.
I have used the multiprocessing lib but it didn't work. My solution is to put original training function into multiprocessing.Pool. For sharing embedding layer:
First, I use multiprocessing.Manager.dict(), but it throws Bad message length Error.
Then I tried to pass embedding layer as a parameter to the training function(It is passed by reference in default). It works well with a sample of glove file(10000 lines) but if I use full glove file(glove.42B.300d.txt), the process exits with 0 without error message, not entering training function.
I also tried use a global variable but the behavior is exactly like pass-to-function.
python python-multiprocessing pytorch
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
add a comment |
I am diving into a deep learning model. That model contains multiple classifier with same word embedding layer and can be trained simultaneously in different process. The original implementation from author uses distinct process, which just launch different training model and train it in a separate process. It is too memory consuming because each process/classifier use a embedding layer which requires load glove file larger than 4GB. I wonder if there is a way to make those process share their embedding layer so that the memory usage will be reduced.
I have used the multiprocessing lib but it didn't work. My solution is to put original training function into multiprocessing.Pool. For sharing embedding layer:
First, I use multiprocessing.Manager.dict(), but it throws Bad message length Error.
Then I tried to pass embedding layer as a parameter to the training function(It is passed by reference in default). It works well with a sample of glove file(10000 lines) but if I use full glove file(glove.42B.300d.txt), the process exits with 0 without error message, not entering training function.
I also tried use a global variable but the behavior is exactly like pass-to-function.
python python-multiprocessing pytorch
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
I am diving into a deep learning model. That model contains multiple classifier with same word embedding layer and can be trained simultaneously in different process. The original implementation from author uses distinct process, which just launch different training model and train it in a separate process. It is too memory consuming because each process/classifier use a embedding layer which requires load glove file larger than 4GB. I wonder if there is a way to make those process share their embedding layer so that the memory usage will be reduced.
I have used the multiprocessing lib but it didn't work. My solution is to put original training function into multiprocessing.Pool. For sharing embedding layer:
First, I use multiprocessing.Manager.dict(), but it throws Bad message length Error.
Then I tried to pass embedding layer as a parameter to the training function(It is passed by reference in default). It works well with a sample of glove file(10000 lines) but if I use full glove file(glove.42B.300d.txt), the process exits with 0 without error message, not entering training function.
I also tried use a global variable but the behavior is exactly like pass-to-function.
python python-multiprocessing pytorch
python python-multiprocessing pytorch
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
asked Nov 23 '18 at 0:53
SongRbSongRb
2613
2613
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439509%2ftrying-to-make-a-global-embedding-layer-provider-in-pytorch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439509%2ftrying-to-make-a-global-embedding-layer-provider-in-pytorch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
