Can arangodb handle 20 billion records?

up vote
1
down vote

favorite

I just want to know, can Arangodb handle 20 billions of records in a single collection?

What is the best way to insert 1 billion records at a time?

asked Nov 19 at 12:20

Yash Pandya

add a comment |

up vote
1
down vote

favorite

I just want to know, can Arangodb handle 20 billions of records in a single collection?

What is the best way to insert 1 billion records at a time?

asked Nov 19 at 12:20

Yash Pandya

add a comment |

up vote
1
down vote

favorite

I just want to know, can Arangodb handle 20 billions of records in a single collection?

What is the best way to insert 1 billion records at a time?

asked Nov 19 at 12:20

Yash Pandya

I just want to know, can Arangodb handle 20 billions of records in a single collection?

What is the best way to insert 1 billion records at a time?

arangodb arangodb-php

asked Nov 19 at 12:20

Yash Pandya

asked Nov 19 at 12:20

Yash Pandya

asked Nov 19 at 12:20

Yash Pandya

asked Nov 19 at 12:20

Yash Pandya

asked Nov 19 at 12:20

Yash Pandya

add a comment |

1 Answer
1

active

oldest

votes

up vote
0
down vote

Of course. The underlying storage engine choice should definitely be RocksDB. How big would every individual document in average be?

The best way would be any sort of import/restore. Is this going to be a cluster or a single server deployment?

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

The individual document can be of size 1 kb. Currently I am using it as a single server and mmfiles as a storage engine.
– Yash Pandya
Nov 20 at 13:40

MMFiles is not an option unless you are looking into buying 20 exabyte of RAM. Memory mapped files performance goes towards zero, when data size including indexes cannot be matched in RAM size.
– Kaveh Vahedipour
Nov 20 at 17:45

Thanks for the reply. You suggested the best way to insert records is import/restore, but to use that my whole data should be ready before insertion,right? Suppose my data is generated at run time using some sort of script, at that time what should I use?
– Yash Pandya
Nov 22 at 14:40

Absolutely. By all means insert all you data.
– Kaveh Vahedipour
Nov 22 at 14:48

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53374517%2fcan-arangodb-handle-20-billion-records%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
0
down vote

Of course. The underlying storage engine choice should definitely be RocksDB. How big would every individual document in average be?

The best way would be any sort of import/restore. Is this going to be a cluster or a single server deployment?

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

The individual document can be of size 1 kb. Currently I am using it as a single server and mmfiles as a storage engine.
– Yash Pandya
Nov 20 at 13:40

MMFiles is not an option unless you are looking into buying 20 exabyte of RAM. Memory mapped files performance goes towards zero, when data size including indexes cannot be matched in RAM size.
– Kaveh Vahedipour
Nov 20 at 17:45

Thanks for the reply. You suggested the best way to insert records is import/restore, but to use that my whole data should be ready before insertion,right? Suppose my data is generated at run time using some sort of script, at that time what should I use?
– Yash Pandya
Nov 22 at 14:40

Absolutely. By all means insert all you data.
– Kaveh Vahedipour
Nov 22 at 14:48

add a comment |

up vote
0
down vote

Of course. The underlying storage engine choice should definitely be RocksDB. How big would every individual document in average be?

The best way would be any sort of import/restore. Is this going to be a cluster or a single server deployment?

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

The individual document can be of size 1 kb. Currently I am using it as a single server and mmfiles as a storage engine.
– Yash Pandya
Nov 20 at 13:40

MMFiles is not an option unless you are looking into buying 20 exabyte of RAM. Memory mapped files performance goes towards zero, when data size including indexes cannot be matched in RAM size.
– Kaveh Vahedipour
Nov 20 at 17:45

Thanks for the reply. You suggested the best way to insert records is import/restore, but to use that my whole data should be ready before insertion,right? Suppose my data is generated at run time using some sort of script, at that time what should I use?
– Yash Pandya
Nov 22 at 14:40

Absolutely. By all means insert all you data.
– Kaveh Vahedipour
Nov 22 at 14:48

add a comment |

up vote
0
down vote

Of course. The underlying storage engine choice should definitely be RocksDB. How big would every individual document in average be?

The best way would be any sort of import/restore. Is this going to be a cluster or a single server deployment?

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

Of course. The underlying storage engine choice should definitely be RocksDB. How big would every individual document in average be?

The best way would be any sort of import/restore. Is this going to be a cluster or a single server deployment?

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

answered Nov 20 at 13:14

Kaveh Vahedipour

2,1601416

The individual document can be of size 1 kb. Currently I am using it as a single server and mmfiles as a storage engine.
– Yash Pandya
Nov 20 at 13:40

MMFiles is not an option unless you are looking into buying 20 exabyte of RAM. Memory mapped files performance goes towards zero, when data size including indexes cannot be matched in RAM size.
– Kaveh Vahedipour
Nov 20 at 17:45

Thanks for the reply. You suggested the best way to insert records is import/restore, but to use that my whole data should be ready before insertion,right? Suppose my data is generated at run time using some sort of script, at that time what should I use?
– Yash Pandya
Nov 22 at 14:40

Absolutely. By all means insert all you data.
– Kaveh Vahedipour
Nov 22 at 14:48

add a comment |

The individual document can be of size 1 kb. Currently I am using it as a single server and mmfiles as a storage engine.
– Yash Pandya
Nov 20 at 13:40

MMFiles is not an option unless you are looking into buying 20 exabyte of RAM. Memory mapped files performance goes towards zero, when data size including indexes cannot be matched in RAM size.
– Kaveh Vahedipour
Nov 20 at 17:45

Thanks for the reply. You suggested the best way to insert records is import/restore, but to use that my whole data should be ready before insertion,right? Suppose my data is generated at run time using some sort of script, at that time what should I use?
– Yash Pandya
Nov 22 at 14:40

Absolutely. By all means insert all you data.
– Kaveh Vahedipour
Nov 22 at 14:48

The individual document can be of size 1 kb. Currently I am using it as a single server and mmfiles as a storage engine.
– Yash Pandya
Nov 20 at 13:40

MMFiles is not an option unless you are looking into buying 20 exabyte of RAM. Memory mapped files performance goes towards zero, when data size including indexes cannot be matched in RAM size.
– Kaveh Vahedipour
Nov 20 at 17:45

Thanks for the reply. You suggested the best way to insert records is import/restore, but to use that my whole data should be ready before insertion,right? Suppose my data is generated at run time using some sort of script, at that time what should I use?
– Yash Pandya
Nov 22 at 14:40

Absolutely. By all means insert all you data.
– Kaveh Vahedipour
Nov 22 at 14:48

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Argthtjtr