Trying to setup consumer and producer in Kafka
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 at 15:03
cricket_007
79.2k1142109
asked Nov 20 at 6:27
Guy Segal
1901520
add a comment |
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 at 15:03
cricket_007
79.2k1142109
asked Nov 20 at 6:27
Guy Segal
1901520
add a comment |
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 at 15:03
cricket_007
79.2k1142109
asked Nov 20 at 6:27
Guy Segal
1901520
I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka
I'm using debug: 'all' mode, this is what I get from the logs:
Producer: test [0]: MessageSet with 1 message(s) delivered
Consumer: Fetch topic test [0] at offset 38 (v2)
The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.
However, for some reason I don't get the message itself in the consumer.
This event is never called:
consumer.on('data', function(m) {
console.log("consumed", m)
});
I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:
https://github.com/guysegal/kafka-example
Specifically, this is the consumer code:
https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts
and the producer code:
https://github.com/guysegal/kafka-example/blob/master/src/producer.ts
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
javascript node.js apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 20 at 15:03
cricket_007
79.2k1142109
asked Nov 20 at 6:27
Guy Segal
1901520
edited Nov 20 at 15:03
cricket_007
79.2k1142109
asked Nov 20 at 6:27
Guy Segal
1901520
edited Nov 20 at 15:03
cricket_007
79.2k1142109
edited Nov 20 at 15:03
cricket_007
79.2k1142109
edited Nov 20 at 15:03
cricket_007
79.2k1142109
79.2k1142109
asked Nov 20 at 6:27
Guy Segal
1901520
asked Nov 20 at 6:27
Guy Segal
1901520
asked Nov 20 at 6:27
Guy Segal
1901520
1901520
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 at 10:23
bittu
332110
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53387406%2ftrying-to-setup-consumer-and-producer-in-kafka%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 at 10:23
bittu
332110
add a comment |
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 at 10:23
bittu
332110
add a comment |
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 at 10:23
bittu
332110
You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.
auto.offset.reset
This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.
The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.
While in case of 'earliest' consumer starts from first available message on topic.
answered Nov 20 at 10:23
bittu
332110
edited Nov 20 at 10:33
answered Nov 20 at 10:23
bittu
332110
answered Nov 20 at 10:23
bittu
332110
answered Nov 20 at 10:23
bittu
332110
332110
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53387406%2ftrying-to-setup-consumer-and-producer-in-kafka%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
