Hi , i have installed filebeat but i have problem.
I need to send only raw content of log file to kafka/logstash without additional information from beats. How to do that?
filebeat config file
filebeat.prospectors: - input_type: log paths: - /var/log/nginx/api.access.log document_type: api_event_beats output.kafka: # initial brokers for reading cluster metadata hosts: ["ny-cluster1:9092", "ny-cluster3:9092", "ny-cluster5:9092", "ny-cluster7:9092", "ny-cluster9:9092"]
# message topic selection + partitioning topic: 'api_events_beats' partition.round_robin: reachable_only: false required_acks: 1 compression: 'none' max_message_bytes: 1000000 worker: 8
when i consume topic i see this. I do need json format.
{"@timestamp":"2016-12-14T07:53:41.393Z","beat":{"hostname":"ny-front4.test.loc","name":"ny-front4.test.loc","version":"5.1.1"},"input_type":"log","message":"ny-front4.test.loc::200::2739::0.012::181.104.24.228::/settings::-::test/3.0::andy-16fddc3a-74f1-4ff5-a3d7-9515504b9232::ar:
:android::302::GET::https::MISS::0.012::0.012::200::3114::14/Dec/2016:07:53:40 +0000::172.16.44.46:3001::-::-::-::6.0::com.test.studio::/settings?onAppStartup=true\u0026app=com.test.studio\u0026type=settings\u0026market=google::application/json; charset=utf-8","offset":2488613629,"source":"/var/log/nginx/api.access.log","type":"api_event_beats"}
but i need to receive this
nginx: ny-front1.test.loc::200::2680::0.000::113.210.109.234::/apps.json::-::test/3.0::andy-ae9317ce-7130-45a8-bf87-dd2d7be4fa3a::my::en::android::248::GET::https::HIT::-::-::-::3055::14/Dec/2016:08:10:10 +0000::-::-::-::-::4.3::com.test.studio::/apps.json?onAppStartup=true&type=props&app=com.test.studio&market=google::application/json; charset=utf-8