Filebeat to kafka message not in order


(Manh Tien Bui) #1

Hi all,

I have a problem: Message in kafka not same formal !

  • My filebeat config:

###################### Filebeat Configuration Example #########################
#=========================== Filebeat prospectors =============================
filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/tongdai1.log
#================================ General =====================================
name: tongdai1

#-------------------------- logstash output ------------------------------
#output.logstash:
 # #Array of hosts to connect to.
  #hosts: ["172.17.103.227:5045"]
  #bulk_max_size: 1000
#---------------------------- Kafka ------------------------------------------
output.kafka:
  hosts: ["172.17.103.226:9092", "172.17.103.228:9092"]
  topic: LOGTEST1
  partition.round_robin:
     reachable_only: false

  required_acks: 1
#  compression: gzip
  max_message_bytes: 1000000
  bulk_max_size: 2048

My message:

2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc10
2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc11
2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc13

When I send to kafka:

{"@timestamp":"2018-05-15T01:35:53.623Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"},"offset":1046,"message":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc10","beat":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"},"source":"/var/log/tongdai1.log"}
{"@timestamp":"2018-05-15T01:35:53.624Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"},"message":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc11","beat":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"},"source":"/var/log/tongdai1.log","offset":1464}
{"@timestamp":"2018-05-15T01:35:59.624Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"},"beat":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"},"source":"/var/log/tongdai1.log","offset":1882,"message":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx_extension_helper: -- Executing [monitor_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc13"}

Please help me !.

Thanks


(ruflin) #2

I'm not sure I fully understand the issue. Are you concerned because in Kafka it is escaped? Or because it's now a json document with additional information?


(Manh Tien Bui) #3

Dear Ruflin,
I checked Filebeat by debug command and I saw:
The first message send out with order:
source --> beat --> @version --> @timestamp --> message --> offset --> host --> tags.

{"source":"/var/log/tongdai.log","beat":{"hostname":"mcr-dev-esb-02","version":"6.1.2","name":"tongdai1"},"@version":"1","@timestamp":"2018-05-25T08:59:42.128Z","message":"2018-05-25 15:59:41 MCR-PrimaryPBX DEB VERBOSE[10864][C-00027429]: pbx.c:4891 in pbx_extension_helper: -- Executing [s@macro-dialout-trunk:19] ExecIf("SIP/2083-0004ac21", "1?Set(CONNECTEDLINE(num,i)=0937953204)") in new stack","offset":4500563664,"host":"mcr-dev-esb-02","tags":["beats_input_codec_plain_applied"]}

The second message send out with order:
beat --> source --> @version --> @timestamp --> message --> offset --> host --> tags

{"beat":{"hostname":"mcr-dev-esb-02","version":"6.1.2","name":"tongdai1"},"source":"/var/log/tongdai.log","@version":"1","@timestamp":"2018-05-25T08:59:42.128Z","message":"2018-05-25 15:59:41 MCR-PrimaryPBX DEB VERBOSE[10864][C-00027429]: app_dial.c:2625 in dial_exec_full: -- Called SIP/ttthn_gsm03/0937953204","offset":4500564728,"host":"mcr-dev-esb-02","tags":["beats_input_codec_plain_applied"]}

I saw order it not same. Why ?. I want to config filebeat send struct message in order. Can you help me?

Thanks.


(ruflin) #4

Now I got it. You are worried about the order of the keys inside the json document. TBH I don't really know why the order would change but when using json it should not matter. Can you share a bit of background on what issues that causes on your end.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.