How to reduce the memory use during building index in pouchDB?
I am using pouchDB in indexdb inside Android webview. Below is the command I used to build the index the first time after syncing database.
db.query('my_index', {
startkey: ["1111"],
endkey: ["1111", {}],
include_docs: false
});
it works well until the database becomes large. It consumes so much memory that it crash the android app. I wonder is there another way to build the index as batch steps? Like I build the index inside a loop, create index for 1000 documents each time.
If it doesn't work, whether I can build the index on couchdb side then sync to browser? Does couchdb support sync index?
Based on the above code, what fields are indexed? Does above code build index on every fields in the database?
couchdb pouchdb
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
add a comment |
I am using pouchDB in indexdb inside Android webview. Below is the command I used to build the index the first time after syncing database.
db.query('my_index', {
startkey: ["1111"],
endkey: ["1111", {}],
include_docs: false
});
it works well until the database becomes large. It consumes so much memory that it crash the android app. I wonder is there another way to build the index as batch steps? Like I build the index inside a loop, create index for 1000 documents each time.
If it doesn't work, whether I can build the index on couchdb side then sync to browser? Does couchdb support sync index?
Based on the above code, what fields are indexed? Does above code build index on every fields in the database?
couchdb pouchdb
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
I don't believe there's any way to do what you're requesting. Unless there's a memory leak or other bug, the only reason it should use a lot of memory when doing the index build, is to hold the actual index in memory. And batching, or building the index elsewhere, wouldn't solve that problem--I expect it would still need to load the index into memory to update the index. How many documents are in your database? The best solution might be to segment your database somehow, so that you have a larger number of smaller indexes.
– Flimzy
Nov 23 '18 at 8:12
The total number of document on database is about 18000. Can I build the index on server side and push to client site?
– Zhao Yi
Nov 23 '18 at 9:03
No, there is no way to synchronize indexes.
– Flimzy
Nov 23 '18 at 10:43
18000 sounds tiny. Let's suppose the device has 200MB free memory. Each individual index record would need to use more than 10kb of memory to crash it, right? Are you sure there isn't something else going on?
– Martin Bramwell
Nov 23 '18 at 11:49
I found the problem is the design doc is not synced to pouchdb from couchdb server. Can I set design doc on couchdb side and sync them to pouchdb?
– Zhao Yi
Nov 28 '18 at 6:12
add a comment |
I am using pouchDB in indexdb inside Android webview. Below is the command I used to build the index the first time after syncing database.
db.query('my_index', {
startkey: ["1111"],
endkey: ["1111", {}],
include_docs: false
});
it works well until the database becomes large. It consumes so much memory that it crash the android app. I wonder is there another way to build the index as batch steps? Like I build the index inside a loop, create index for 1000 documents each time.
If it doesn't work, whether I can build the index on couchdb side then sync to browser? Does couchdb support sync index?
Based on the above code, what fields are indexed? Does above code build index on every fields in the database?
couchdb pouchdb
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
I am using pouchDB in indexdb inside Android webview. Below is the command I used to build the index the first time after syncing database.
db.query('my_index', {
startkey: ["1111"],
endkey: ["1111", {}],
include_docs: false
});
it works well until the database becomes large. It consumes so much memory that it crash the android app. I wonder is there another way to build the index as batch steps? Like I build the index inside a loop, create index for 1000 documents each time.
If it doesn't work, whether I can build the index on couchdb side then sync to browser? Does couchdb support sync index?
Based on the above code, what fields are indexed? Does above code build index on every fields in the database?
couchdb pouchdb
couchdb pouchdb
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
edited Nov 23 '18 at 9:14
Zhao Yi
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
asked Nov 23 '18 at 3:27
Zhao YiZhao Yi
5,6621556131
5,6621556131
I don't believe there's any way to do what you're requesting. Unless there's a memory leak or other bug, the only reason it should use a lot of memory when doing the index build, is to hold the actual index in memory. And batching, or building the index elsewhere, wouldn't solve that problem--I expect it would still need to load the index into memory to update the index. How many documents are in your database? The best solution might be to segment your database somehow, so that you have a larger number of smaller indexes.
– Flimzy
Nov 23 '18 at 8:12
The total number of document on database is about 18000. Can I build the index on server side and push to client site?
– Zhao Yi
Nov 23 '18 at 9:03
No, there is no way to synchronize indexes.
– Flimzy
Nov 23 '18 at 10:43
18000 sounds tiny. Let's suppose the device has 200MB free memory. Each individual index record would need to use more than 10kb of memory to crash it, right? Are you sure there isn't something else going on?
– Martin Bramwell
Nov 23 '18 at 11:49
I found the problem is the design doc is not synced to pouchdb from couchdb server. Can I set design doc on couchdb side and sync them to pouchdb?
– Zhao Yi
Nov 28 '18 at 6:12
add a comment |
I don't believe there's any way to do what you're requesting. Unless there's a memory leak or other bug, the only reason it should use a lot of memory when doing the index build, is to hold the actual index in memory. And batching, or building the index elsewhere, wouldn't solve that problem--I expect it would still need to load the index into memory to update the index. How many documents are in your database? The best solution might be to segment your database somehow, so that you have a larger number of smaller indexes.
– Flimzy
Nov 23 '18 at 8:12
The total number of document on database is about 18000. Can I build the index on server side and push to client site?
– Zhao Yi
Nov 23 '18 at 9:03
No, there is no way to synchronize indexes.
– Flimzy
Nov 23 '18 at 10:43
18000 sounds tiny. Let's suppose the device has 200MB free memory. Each individual index record would need to use more than 10kb of memory to crash it, right? Are you sure there isn't something else going on?
– Martin Bramwell
Nov 23 '18 at 11:49
I found the problem is the design doc is not synced to pouchdb from couchdb server. Can I set design doc on couchdb side and sync them to pouchdb?
– Zhao Yi
Nov 28 '18 at 6:12
I don't believe there's any way to do what you're requesting. Unless there's a memory leak or other bug, the only reason it should use a lot of memory when doing the index build, is to hold the actual index in memory. And batching, or building the index elsewhere, wouldn't solve that problem--I expect it would still need to load the index into memory to update the index. How many documents are in your database? The best solution might be to segment your database somehow, so that you have a larger number of smaller indexes.
– Flimzy
Nov 23 '18 at 8:12
I don't believe there's any way to do what you're requesting. Unless there's a memory leak or other bug, the only reason it should use a lot of memory when doing the index build, is to hold the actual index in memory. And batching, or building the index elsewhere, wouldn't solve that problem--I expect it would still need to load the index into memory to update the index. How many documents are in your database? The best solution might be to segment your database somehow, so that you have a larger number of smaller indexes.
– Flimzy
Nov 23 '18 at 8:12
The total number of document on database is about 18000. Can I build the index on server side and push to client site?
– Zhao Yi
Nov 23 '18 at 9:03
The total number of document on database is about 18000. Can I build the index on server side and push to client site?
– Zhao Yi
Nov 23 '18 at 9:03
No, there is no way to synchronize indexes.
– Flimzy
Nov 23 '18 at 10:43
No, there is no way to synchronize indexes.
– Flimzy
Nov 23 '18 at 10:43
18000 sounds tiny. Let's suppose the device has 200MB free memory. Each individual index record would need to use more than 10kb of memory to crash it, right? Are you sure there isn't something else going on?
– Martin Bramwell
Nov 23 '18 at 11:49
18000 sounds tiny. Let's suppose the device has 200MB free memory. Each individual index record would need to use more than 10kb of memory to crash it, right? Are you sure there isn't something else going on?
– Martin Bramwell
Nov 23 '18 at 11:49
I found the problem is the design doc is not synced to pouchdb from couchdb server. Can I set design doc on couchdb side and sync them to pouchdb?
– Zhao Yi
Nov 28 '18 at 6:12
I found the problem is the design doc is not synced to pouchdb from couchdb server. Can I set design doc on couchdb side and sync them to pouchdb?
– Zhao Yi
Nov 28 '18 at 6:12
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53440323%2fhow-to-reduce-the-memory-use-during-building-index-in-pouchdb%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53440323%2fhow-to-reduce-the-memory-use-during-building-index-in-pouchdb%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I don't believe there's any way to do what you're requesting. Unless there's a memory leak or other bug, the only reason it should use a lot of memory when doing the index build, is to hold the actual index in memory. And batching, or building the index elsewhere, wouldn't solve that problem--I expect it would still need to load the index into memory to update the index. How many documents are in your database? The best solution might be to segment your database somehow, so that you have a larger number of smaller indexes.
– Flimzy
Nov 23 '18 at 8:12
The total number of document on database is about 18000. Can I build the index on server side and push to client site?
– Zhao Yi
Nov 23 '18 at 9:03
No, there is no way to synchronize indexes.
– Flimzy
Nov 23 '18 at 10:43
18000 sounds tiny. Let's suppose the device has 200MB free memory. Each individual index record would need to use more than 10kb of memory to crash it, right? Are you sure there isn't something else going on?
– Martin Bramwell
Nov 23 '18 at 11:49
I found the problem is the design doc is not synced to pouchdb from couchdb server. Can I set design doc on couchdb side and sync them to pouchdb?
– Zhao Yi
Nov 28 '18 at 6:12