# Filebeat to kafka message not in order **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 15, 2018, 3:07am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-message-not-in-order/131836 "2018-05-15T03:07:00Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Manh\_Tien\_Bui](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Manh\_Tien\_Bui](https://discuss.elastic.co/u/Manh_Tien_Bui)\ **Post date:** [May 15, 2018, 3:07am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-message-not-in-order/131836/1 "2018-05-15T03:07:00Z")
Hi all, # I have a problem: Message in kafka not same formal ! - My filebeat config: > ###################### Filebeat Configuration Example ######################### > #=========================== Filebeat prospectors ============================= > filebeat.prospectors: > > ``` > # Each - is a prospector. Most options can be set at the prospector level, so > # you can use different prospectors for various configurations. > # Below are the prospector specific configurations. > > - input_type: log > > # Paths that should be crawled and fetched. Glob based paths. > paths: > - /var/log/tongdai1.log > #================================ General ===================================== > name: tongdai1 > > #-------------------------- logstash output ------------------------------ > #output.logstash: > # #Array of hosts to connect to. > #hosts: ["172.17.103.227:5045"] > #bulk_max_size: 1000 > #---------------------------- Kafka ------------------------------------------ > output.kafka: > hosts: ["172.17.103.226:9092", "172.17.103.228:9092"] > topic: LOGTEST1 > partition.round_robin: > reachable_only: false > > required_acks: 1 > # compression: gzip > max_message_bytes: 1000000 > bulk_max_size: 2048 > > ``` # My message: > 2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [monitor\_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc10 > 2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [monitor\_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc11 > 2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [monitor\_default@sub-ccss:3] Return("SIP/2100-001ff840", "FALSE") in new stack abc13 # When I send to kafka: > {"@timestamp":"2018-05-15T01:35:53.623Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"}," **offset**":1046," **message**":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [monitor\_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc10"," **beat**":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"}," **source**":"/var/log/tongdai1.log"} > {"@timestamp":"2018-05-15T01:35:53.624Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"}," **message**":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [monitor\_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc11"," **beat**":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"}," **source**":"/var/log/tongdai1.log","offset":1464} > {"@timestamp":"2018-05-15T01:35:59.624Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"LOGTEST1"}," **beat**":{"name":"tongdai1","hostname":"mcr-dev-esb-02","version":"6.1.2"}," **source**":"/var/log/tongdai1.log"," **offset**":1882," **message**":"2018-05-09 16:35:49 MCR-PrimaryPBX DEB VERBOSE[3711][C-0010e89b]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [monitor\_default@sub-ccss:3] Return(\"SIP/2100-001ff840\", \"FALSE\") in new stack abc13"} Please help me !. Thanks ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 15, 2018, 1:03pm UTC](https://discuss.elastic.co/t/filebeat-to-kafka-message-not-in-order/131836/2 "2018-05-15T13:03:43Z")
I'm not sure I fully understand the issue. Are you concerned because in Kafka it is escaped? Or because it's now a json document with additional information? ---
**Author:** ![Manh\_Tien\_Bui](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Manh\_Tien\_Bui](https://discuss.elastic.co/u/Manh_Tien_Bui)\ **Post date:** [May 25, 2018, 9:13am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-message-not-in-order/131836/3 "2018-05-25T09:13:07Z")
Dear Ruflin, I checked Filebeat by debug command and I saw: The first message send out with order: source --\> beat --\> @version --\> @timestamp --\> message --\> offset --\> host --\> tags. > {"source":"/var/log/tongdai.log","beat":{"hostname":"mcr-dev-esb-02","version":"6.1.2","name":"tongdai1"},"@version":"1","@timestamp":"2018-05-25T08:59:42.128Z","message":"2018-05-25 15:59:41 MCR-PrimaryPBX DEB VERBOSE[10864][C-00027429]: pbx.c:4891 in pbx\_extension\_helper: -- Executing [s@macro-dialout-trunk:19] ExecIf("SIP/2083-0004ac21", "1?Set(CONNECTEDLINE(num,i)=0937953204)") in new stack","offset":4500563664,"host":"mcr-dev-esb-02","tags":["beats\_input\_codec\_plain\_applied"]} The second message send out with order: beat --\> source --\> @version --\> @timestamp --\> message --\> offset --\> host --\> tags > {"beat":{"hostname":"mcr-dev-esb-02","version":"6.1.2","name":"tongdai1"},"source":"/var/log/tongdai.log","@version":"1","@timestamp":"2018-05-25T08:59:42.128Z","message":"2018-05-25 15:59:41 MCR-PrimaryPBX DEB VERBOSE[10864][C-00027429]: app\_dial.c:2625 in dial\_exec\_full: -- Called SIP/ttthn\_gsm03/0937953204","offset":4500564728,"host":"mcr-dev-esb-02","tags":["beats\_input\_codec\_plain\_applied"]} I saw order it not same. Why ?. I want to config filebeat send struct message in order. Can you help me? Thanks. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 25, 2018, 12:52pm UTC](https://discuss.elastic.co/t/filebeat-to-kafka-message-not-in-order/131836/4 "2018-05-25T12:52:52Z")
Now I got it. You are worried about the order of the keys inside the json document. TBH I don't really know why the order would change but when using json it should not matter. Can you share a bit of background on what issues that causes on your end. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2018, 12:52pm UTC](https://discuss.elastic.co/t/filebeat-to-kafka-message-not-in-order/131836/5 "2018-06-22T12:52:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.