Argthtjtr

I have an issue with Filebeat when I try to send data logs to 2 Kafka nodes at the same time

The following is the Output Kafka section of the filebeat.yml file:

output.kafka:

  enabled: true

  hosts: [ "192.168.xxx.xx:9092", "192.168.zzz.zz:9092" ]

  topic: "syslog"

  timeout: 30s

  max_message_bytes: 1000000

Both kafka services are running, but only the second node get data. I mean Only the Kafka node 192.168.zzz.zz get the data Filebeat have sent it.

If I exchange the IP adressess, occurs that the second IP address get the data log.

Why occurs that ? What other configurations are need to implement this use case? I need send the data to both kafka outputs.

Assuming both broker urls form the same cluster, only one address is used to bootstrap the rest of the cluster. If one of those addresses is not reachable, then the other is selected.

If Filebeat is creating messages with null keys, then messages should be evenly spread across partitions within the specified kafka topic for the cluster that is being connected to.

Data is only sent to the leader for the calculated partition (based on the message key), therefore, a single message cannot be sent to "two nodes (of the same cluster) at the same time". Also, if you had more than those two servers in the Kafka cluster, the one that gets the data could one that is not part of the those addresses you've listed.

I don't think Filebeat can output to more than one unique Kafka cluster at once, at least least not within a single output.kafka section. Logstash might work better for that use case