Hi all,
I have a problem: Message in kafka not same formal !
- My filebeat config:
###################### Filebeat Configuration Example #########################
#=========================== Filebeat prospectors =============================
filebeat.prospectors:# Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - input_type: log # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/tongdai1.log #================================ General ===================================== name: tongdai1 #-------------------------- logstash output ------------------------------ #output.logstash: # #Array of hosts to connect to. #hosts: ["172.17.103.227:5045"] #bulk_max_size: 1000 #---------------------------- Kafka ------------------------------------------ output.kafka: hosts: ["172.17.103.226:9092", "172.17.103.228:9092"] topic: LOGTEST1 partition.round_robin: reachable_only: false required_acks: 1 # compression: gzip max_message_bytes: 1000000 bulk_max_size: 2048
My message:
2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc10
2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc11
2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc13
When I send to kafka:
{"@timestamp":"2018-05-15T01:35:53.623Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"},"offset":1046,"message":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc10","beat":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"},"source":"/var/log/tongdai1.log"}
{"@timestamp":"2018-05-15T01:35:53.624Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"},"message":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc11","beat":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"},"source":"/var/log/tongdai1.log","offset":1464}
{"@timestamp":"2018-05-15T01:35:59.624Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"},"beat":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"},"source":"/var/log/tongdai1.log","offset":1882,"message":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc13"}
Please help me !.
Thanks