how can I insert a Tensor into another Tensor in pytorch
I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.
I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.
I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.
I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.
A = Variable(torch.zeros(batch_size, step, vec_size))
What I'm doing is like:
for i in range(batch_size):
pos = P[i]
A[i][pos] = A[i][pos] + B[i]
But I get an Error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Then, I make a clone of A each inside the loop:
for i in range(batch_size):
A_clone = A.clone()
pos = P[i]
A_clone[i][pos] = A_clone[i][pos] + B[i]
This is very slow for autograd, I wonder if there any better solutions? Thank you.
pytorch
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
asked Nov 22 '18 at 21:15
TongTong
32
add a comment |
I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.
I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.
I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.
I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.
A = Variable(torch.zeros(batch_size, step, vec_size))
What I'm doing is like:
for i in range(batch_size):
pos = P[i]
A[i][pos] = A[i][pos] + B[i]
But I get an Error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Then, I make a clone of A each inside the loop:
for i in range(batch_size):
A_clone = A.clone()
pos = P[i]
A_clone[i][pos] = A_clone[i][pos] + B[i]
This is very slow for autograd, I wonder if there any better solutions? Thank you.
pytorch
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
asked Nov 22 '18 at 21:15
TongTong
32
add a comment |
I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.
I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.
I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.
I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.
A = Variable(torch.zeros(batch_size, step, vec_size))
What I'm doing is like:
for i in range(batch_size):
pos = P[i]
A[i][pos] = A[i][pos] + B[i]
But I get an Error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Then, I make a clone of A each inside the loop:
for i in range(batch_size):
A_clone = A.clone()
pos = P[i]
A_clone[i][pos] = A_clone[i][pos] + B[i]
This is very slow for autograd, I wonder if there any better solutions? Thank you.
pytorch
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
asked Nov 22 '18 at 21:15
TongTong
32
I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.
I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.
I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.
I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.
A = Variable(torch.zeros(batch_size, step, vec_size))
What I'm doing is like:
for i in range(batch_size):
pos = P[i]
A[i][pos] = A[i][pos] + B[i]
But I get an Error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Then, I make a clone of A each inside the loop:
for i in range(batch_size):
A_clone = A.clone()
pos = P[i]
A_clone[i][pos] = A_clone[i][pos] + B[i]
This is very slow for autograd, I wonder if there any better solutions? Thank you.
pytorch
pytorch
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
asked Nov 22 '18 at 21:15
TongTong
32
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
asked Nov 22 '18 at 21:15
TongTong
32
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
edited Nov 22 '18 at 21:57
Umang Gupta
3,51111739
3,51111739
asked Nov 22 '18 at 21:15
TongTong
32
asked Nov 22 '18 at 21:15
TongTong
32
asked Nov 22 '18 at 21:15
TongTong
32
32
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You can use a mask instead of cloning.
See the code below
# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()
# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0
# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)
This way you avoid using for loops and cloning as well.
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with theunsqueezecall.
– user2268997
Dec 8 '18 at 13:28
1
@user2268997unsqueezecall makesbof shape[batch_size, 1, vec_size]. Note that shape ofbwas[batch_size, vec_size]and expand broadcasts it to make it same size as mask so as to be able to multiply
– Umang Gupta
Dec 8 '18 at 17:39
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438062%2fhow-can-i-insert-a-tensor-into-another-tensor-in-pytorch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can use a mask instead of cloning.
See the code below
# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()
# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0
# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)
This way you avoid using for loops and cloning as well.
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with theunsqueezecall.
– user2268997
Dec 8 '18 at 13:28
1
@user2268997unsqueezecall makesbof shape[batch_size, 1, vec_size]. Note that shape ofbwas[batch_size, vec_size]and expand broadcasts it to make it same size as mask so as to be able to multiply
– Umang Gupta
Dec 8 '18 at 17:39
add a comment |
You can use a mask instead of cloning.
See the code below
# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()
# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0
# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)
This way you avoid using for loops and cloning as well.
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with theunsqueezecall.
– user2268997
Dec 8 '18 at 13:28
1
@user2268997unsqueezecall makesbof shape[batch_size, 1, vec_size]. Note that shape ofbwas[batch_size, vec_size]and expand broadcasts it to make it same size as mask so as to be able to multiply
– Umang Gupta
Dec 8 '18 at 17:39
add a comment |
You can use a mask instead of cloning.
See the code below
# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()
# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0
# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)
This way you avoid using for loops and cloning as well.
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
You can use a mask instead of cloning.
See the code below
# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()
# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0
# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)
This way you avoid using for loops and cloning as well.
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
answered Nov 22 '18 at 22:11
Umang GuptaUmang Gupta
3,51111739
3,51111739
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with theunsqueezecall.
– user2268997
Dec 8 '18 at 13:28
1
@user2268997unsqueezecall makesbof shape[batch_size, 1, vec_size]. Note that shape ofbwas[batch_size, vec_size]and expand broadcasts it to make it same size as mask so as to be able to multiply
– Umang Gupta
Dec 8 '18 at 17:39
add a comment |
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with theunsqueezecall.
– user2268997
Dec 8 '18 at 13:28
1
@user2268997unsqueezecall makesbof shape[batch_size, 1, vec_size]. Note that shape ofbwas[batch_size, vec_size]and expand broadcasts it to make it same size as mask so as to be able to multiply
– Umang Gupta
Dec 8 '18 at 17:39
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with the
unsqueeze call.– user2268997
Dec 8 '18 at 13:28
Hi Umang. Thanks for the great answer. Would you mind elaborating on that last line a bit? I am confused with the
unsqueeze call.– user2268997
Dec 8 '18 at 13:28
1
1
@user2268997
unsqueeze call makes b of shape [batch_size, 1, vec_size]. Note that shape of b was [batch_size, vec_size] and expand broadcasts it to make it same size as mask so as to be able to multiply– Umang Gupta
Dec 8 '18 at 17:39
@user2268997
unsqueeze call makes b of shape [batch_size, 1, vec_size]. Note that shape of b was [batch_size, vec_size] and expand broadcasts it to make it same size as mask so as to be able to multiply– Umang Gupta
Dec 8 '18 at 17:39
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438062%2fhow-can-i-insert-a-tensor-into-another-tensor-in-pytorch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
