Filebeat sends json to kafka instead of raw content of log file

Elastic Stack Beats
1

Hi , i have installed filebeat but i have problem.
I need to send only raw content of log file to kafka/logstash without additional information from beats. How to do that?

filebeat config file

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/api.access.log
  document_type: api_event_beats
output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["ny-cluster1:9092", "ny-cluster3:9092", "ny-cluster5:9092", "ny-cluster7:9092", "ny-cluster9:9092"]

  # message topic selection + partitioning
  topic: 'api_events_beats'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: 'none'
  max_message_bytes: 1000000
  worker: 8

when i consume topic i see this. I do need json format.

{"@timestamp":"2016-12-14T07:53:41.393Z","beat":{"hostname":"ny-front4.test.loc","name":"ny-front4.test.loc","version":"5.1.1"},"input_type":"log","message":"ny-front4.test.loc::200::2739::0.012::181.104.24.228::/settings::-::test/3.0::andy-16fddc3a-74f1-4ff5-a3d7-9515504b9232::ar::es::android::302::GET::https::MISS::0.012::0.012::200::3114::14/Dec/2016:07:53:40 +0000::172.16.44.46:3001::-::-::-::6.0::com.test.studio::/settings?onAppStartup=true\u0026app=com.test.studio\u0026type=settings\u0026market=google::application/json; charset=utf-8","offset":2488613629,"source":"/var/log/nginx/api.access.log","type":"api_event_beats"}

but i need to receive this

nginx: ny-front1.test.loc::200::2680::0.000::113.210.109.234::/apps.json::-::test/3.0::andy-ae9317ce-7130-45a8-bf87-dd2d7be4fa3a::my::en::android::248::GET::https::HIT::-::-::-::3055::14/Dec/2016:08:10:10 +0000::-::-::-::-::4.3::com.test.studio::/apps.json?onAppStartup=true&type=props&app=com.test.studio&market=google::application/json; charset=utf-8

2

This is currently not supported. There is an open PR introducing this feature.

3

Steffen, maybe you now when this feature will be implemented?

4

I have no idea if and when this feature will be merged. The original author didn't answer the code review yet.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.