Import Contact Issue
I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.
Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.
I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?
I verified the processor, free memory, file size, etc are all way under utilized.
Rememdy is kill all user/application processes, repair database and continue forward.
But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.
Thx
import
asked 2 days ago
drummondislebsd
361
add a comment |
I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.
Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.
I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?
I verified the processor, free memory, file size, etc are all way under utilized.
Rememdy is kill all user/application processes, repair database and continue forward.
But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.
Thx
import
asked 2 days ago
drummondislebsd
361
I've rarely had good luck importing more than 500-800 contacts in one shot with CiviCRM. Have you tried splitting your import into smaller batches?
– Lesley Carter - BackOffice
2 days ago
add a comment |
I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.
Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.
I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?
I verified the processor, free memory, file size, etc are all way under utilized.
Rememdy is kill all user/application processes, repair database and continue forward.
But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.
Thx
import
asked 2 days ago
drummondislebsd
361
I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.
Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.
I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?
I verified the processor, free memory, file size, etc are all way under utilized.
Rememdy is kill all user/application processes, repair database and continue forward.
But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.
Thx
import
import
asked 2 days ago
drummondislebsd
361
asked 2 days ago
drummondislebsd
361
asked 2 days ago
drummondislebsd
361
asked 2 days ago
drummondislebsd
361
asked 2 days ago
drummondislebsd
361
361
I've rarely had good luck importing more than 500-800 contacts in one shot with CiviCRM. Have you tried splitting your import into smaller batches?
– Lesley Carter - BackOffice
2 days ago
add a comment |
I've rarely had good luck importing more than 500-800 contacts in one shot with CiviCRM. Have you tried splitting your import into smaller batches?
– Lesley Carter - BackOffice
2 days ago
I've rarely had good luck importing more than 500-800 contacts in one shot with CiviCRM. Have you tried splitting your import into smaller batches?
– Lesley Carter - BackOffice
2 days ago
I've rarely had good luck importing more than 500-800 contacts in one shot with CiviCRM. Have you tried splitting your import into smaller batches?
– Lesley Carter - BackOffice
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.
Past that:
One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.
Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).
I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.
If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "605"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcivicrm.stackexchange.com%2fquestions%2f27859%2fimport-contact-issue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.
Past that:
One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.
Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).
I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.
If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
add a comment |
You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.
Past that:
One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.
Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).
I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.
If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
add a comment |
You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.
Past that:
One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.
Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).
I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.
If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.
Past that:
One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.
Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).
I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.
If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
answered 2 days ago
Jon G - Megaphone Tech
25.6k11767
25.6k11767
add a comment |
add a comment |
Thanks for contributing an answer to CiviCRM Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcivicrm.stackexchange.com%2fquestions%2f27859%2fimport-contact-issue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I've rarely had good luck importing more than 500-800 contacts in one shot with CiviCRM. Have you tried splitting your import into smaller batches?
– Lesley Carter - BackOffice
2 days ago