Libbeat.kafka.published_but_not_acked_events

Elastic Stack Beats
1

my filebeat config file:
filebeat:
prospectors:
-
paths:
- /var/log/xyz.log
document_type: log
fields: {log_type: log1}
-
paths:
- /var/log/abc.log
document_type: log
fields: {log_type: log2}
output:
kafka:
hosts: ["localhost:9092"]
topic: "%{[log_type]}"

Now I am trying to create two different kafka topics for 2 separate files but unable to do so.
My filebeat is running but keeps on printing the below lines-:

INFO Non-zero metrics in the last 30s: libbeat.kafka.published_but_not_acked_events=890384 libbeat.kafka.call_count.PublishEvents=222596

Any help on this would be appreciated.

2

Could you have a look at the log file? There should be quite a few error messages. Please share this part.

3

Thanks for reply. The log file is ok bçoz if i give static name for topic, data is coming to kafka topic.
ex. topic: "test"
This way the data is coming at kafka topic

But when i try to pass dynamic value for topic name (as given in original post), I do not recieve any data nor any kind of topic is created at kafka.

Even though if you want to have a look at log file its like
"nov 9 16:23:45 10.0.2.9 INFO [application level message]",

Lines in same format are written into log file every time
Just wanted to know is the configuration of filebeat correct?????

4

Found the solution to it. The fields defined under paths are local and therefore cannot be accessed outside that path. To make the fields global we need to add following lines after fields parameter
fields_under_root: true

This makes the fields key accessible from outside the path as well.
Hope I understood this correctly. Please let me know if I wrote anything wrong.

5

you don't need fields_under_root. Using topic: "%{[fields.log_type]}" should do the trick.

6

This topic was automatically closed after 21 days. New replies are no longer allowed.