AWS upload folder to S3 as tar.gz without compressing locally
In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?
For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz
How do I do that without turning it into a tar.gz locally? (I want to save local space, as I don't have much space on my HDD.)
tar amazon-web-services amazon-s3
edited Sep 7 '17 at 16:03
Scott
15.6k113889
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
add a comment |
In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?
For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz
How do I do that without turning it into a tar.gz locally? (I want to save local space, as I don't have much space on my HDD.)
tar amazon-web-services amazon-s3
edited Sep 7 '17 at 16:03
Scott
15.6k113889
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
add a comment |
In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?
For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz
How do I do that without turning it into a tar.gz locally? (I want to save local space, as I don't have much space on my HDD.)
tar amazon-web-services amazon-s3
edited Sep 7 '17 at 16:03
Scott
15.6k113889
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?
For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz
How do I do that without turning it into a tar.gz locally? (I want to save local space, as I don't have much space on my HDD.)
tar amazon-web-services amazon-s3
tar amazon-web-services amazon-s3
edited Sep 7 '17 at 16:03
Scott
15.6k113889
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
edited Sep 7 '17 at 16:03
Scott
15.6k113889
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
edited Sep 7 '17 at 16:03
Scott
15.6k113889
edited Sep 7 '17 at 16:03
Scott
15.6k113889
edited Sep 7 '17 at 16:03
Scott
15.6k113889
15.6k113889
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
asked Sep 7 '17 at 15:00
Michael SamsungMichael Samsung
2612
2612
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.
tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"
Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):
tar c /var/testcreates atararchive out of/var/testand outputs it tostdout...- ...which is read by
gzipfromstdin, and the gzipped file (.tar.gz) is output tostdout... - ...which is read by
aws s3 cp - "s3://tests/test1.tar.gz"fromstdinand sent to S3. The-tells the AWS CLI to copy fromstdin.
This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
1
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
1
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
1
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
|
show 3 more comments
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1248276%2faws-upload-folder-to-s3-as-tar-gz-without-compressing-locally%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.
tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"
Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):
tar c /var/testcreates atararchive out of/var/testand outputs it tostdout...- ...which is read by
gzipfromstdin, and the gzipped file (.tar.gz) is output tostdout... - ...which is read by
aws s3 cp - "s3://tests/test1.tar.gz"fromstdinand sent to S3. The-tells the AWS CLI to copy fromstdin.
This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
1
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
1
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
1
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
|
show 3 more comments
What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.
tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"
Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):
tar c /var/testcreates atararchive out of/var/testand outputs it tostdout...- ...which is read by
gzipfromstdin, and the gzipped file (.tar.gz) is output tostdout... - ...which is read by
aws s3 cp - "s3://tests/test1.tar.gz"fromstdinand sent to S3. The-tells the AWS CLI to copy fromstdin.
This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
1
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
1
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
1
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
|
show 3 more comments
What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.
tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"
Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):
tar c /var/testcreates atararchive out of/var/testand outputs it tostdout...- ...which is read by
gzipfromstdin, and the gzipped file (.tar.gz) is output tostdout... - ...which is read by
aws s3 cp - "s3://tests/test1.tar.gz"fromstdinand sent to S3. The-tells the AWS CLI to copy fromstdin.
This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.
tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"
Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):
tar c /var/testcreates atararchive out of/var/testand outputs it tostdout...- ...which is read by
gzipfromstdin, and the gzipped file (.tar.gz) is output tostdout... - ...which is read by
aws s3 cp - "s3://tests/test1.tar.gz"fromstdinand sent to S3. The-tells the AWS CLI to copy fromstdin.
This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
edited Sep 8 '17 at 1:34
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
answered Sep 7 '17 at 15:18
BobBob
45.4k20137172
45.4k20137172
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
1
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
1
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
1
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
|
show 3 more comments
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
1
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
1
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
1
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
– Tim
Sep 8 '17 at 0:59
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
@Tim ...somehow, I completely missed that. I'll update.
– Bob
Sep 8 '17 at 1:04
1
1
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
@Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
– Bob
Sep 8 '17 at 1:11
1
1
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
– murze
Jun 10 '18 at 9:51
1
1
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
@murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
– Ekevoo
Sep 12 '18 at 17:50
|
show 3 more comments
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1248276%2faws-upload-folder-to-s3-as-tar-gz-without-compressing-locally%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown