Conditional topics not work: since document_type is deprecated, how to use %{[type]} in kafka ouput?

Elastic Stack Beats
1

anyway, this %{[type]} is still valid in the filebeat 6.0.0 docs. but it is not working in program. and when I use fields it is also not working for topics, e.g:
filebeat.prospectors:
...
fields:
log_topic: test1
...
output.kafka:
...
topic: '%{[log_topic]}'

2

Can you share your complete docs (please use the </> button for formatting configs and logs)? The document_type setting should still work. Ithink it's the output hardcoding _type to docs, but document_type sets the type field.

By default the fields setting adds events to the fields namespace. You might have to use topic: '%{[fields.log_topic]}'.

3

the doc is in:

all of them are still recommend %{[type]}. but I test in filebeat 6.0.0, it can not refer to document_type

and thanks for 'topic: '%{[fields.log_topic]}'.' it work for me

4

Could you share your full config and logs? Please format them using </>.

5

document_type was removed in 6.0 after it was deprecated in 5.5: https://github.com/elastic/beats/pull/4204

fields should be used for that now. We should fix our docs here :frowning:

@oldcodeoberyn Want to open a Github issue for it?

6

:sunglasses: originally, I want to open a github issue before I opened this discussion. but it is already 10 day pass, I think you have been fix it already in doc?

7

Hi, this is a config that works for us:

filebeat.prospectors:
- type: log
  paths:
  - /var/log/log1.log
  fields:
    kafka_topic: elk1
  fields_under_root: true
- type: log
  paths:
  - /var/log/log2.log
  fields:
    kafka_topic: elk2
  fields_under_root: true
output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
  topic: '%{[kafka_topic]}'

It should also work without the fields_under_root thing, which would look like below - this is untested, however:

filebeat.prospectors:
- type: log
  paths:
  - /var/log/log1.log
  fields:
    kafka_topic: elk1
- type: log
  paths:
  - /var/log/log2.log
  fields:
    kafka_topic: elk2
output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
  topic: '%{[fields.kafka_topic]}'

I agree with @oldcodeoberyn that this should be fixed soon, it's really misleading and many people are running into it.

8

It't not fixed yet on https://www.elastic.co/guide/en/beats/filebeat/6.1/kafka-output.html#_literal_topic_literal or https://www.elastic.co/guide/en/beats/filebeat/6.0/kafka-output.html#_literal_topic_literal. There's a pull request https://github.com/elastic/beats/pull/5867 .

9

The PR has been merged into the master branch and should show up in the other branches sometime today. We require every PR to be reviewed before it's merged...hence the delay. Thanks for your patience.

10

Cool, it's fixed, thanks

11

This topic was automatically closed after 21 days. New replies are no longer allowed.