Apache Beam RetryTransientErrors/neverRetry does not respect table not found error
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam
asked Nov 20 '18 at 11:44
TameemTameem
1279
add a comment |
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam
asked Nov 20 '18 at 11:44
TameemTameem
1279
add a comment |
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam
asked Nov 20 '18 at 11:44
TameemTameem
1279
Below is the code that I'm using to write data to BigQuery
WriteResult result = formattedData.get(successRows).setCoder(TableRowJsonCoder.of())
.apply("BQ SteamingInserts",BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFormatFunction(new TableRowFormatFn())
.to(new DestinationMapper())
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
.withoutValidation()
.withExtendedErrorInfo());
The code is handling all the schema related issues but when table does not exist in BigQuery, it keeps retrying the inserts indefinitely resulting in pipeline getting stalled.
Below is the error obtained in Dataflow
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"reason" : "notFound"
} ],
"message" : "Not found: Table analytics-and-presentation:trusted_layer_ods.wrong_table",
"status" : "NOT_FOUND"
}
Can someone help?
google-cloud-dataflow apache-beam
google-cloud-dataflow apache-beam
asked Nov 20 '18 at 11:44
TameemTameem
1279
asked Nov 20 '18 at 11:44
TameemTameem
1279
edited Nov 20 '18 at 16:21
Tameem
asked Nov 20 '18 at 11:44
TameemTameem
1279
asked Nov 20 '18 at 11:44
TameemTameem
1279
asked Nov 20 '18 at 11:44
TameemTameem
1279
1279
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392311%2fapache-beam-retrytransienterrors-neverretry-does-not-respect-table-not-found-err%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
add a comment |
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
add a comment |
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
This looks like is an expected behavior as Dataflow will reentry indefinitely in the case of a streaming pipeline. The job would have failed after 4 tries in a batch pipeline.
You must clearly define in your code what you expect to achieve. You can get inspiration from the examples found on the official Google Cloud Platform github page.
With your present code you must make sure you create the table in advance so to avoid this error.
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
answered Nov 21 '18 at 22:32
MonicaPCMonicaPC
1427
1427
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392311%2fapache-beam-retrytransienterrors-neverretry-does-not-respect-table-not-found-err%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
