# Filebeat merged 2 "normal" lines **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 18, 2018, 2:43pm UTC](https://discuss.elastic.co/t/filebeat-merged-2-normal-lines/140580 "2018-07-18T14:43:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ilia\_Matveikin](https://avatars.discourse-cdn.com/v4/letter/i/ecae2f/32.png) [@Ilia\_Matveikin](https://discuss.elastic.co/u/Ilia_Matveikin)\ **Post date:** [July 18, 2018, 2:43pm UTC](https://discuss.elastic.co/t/filebeat-merged-2-normal-lines/140580/1 "2018-07-18T14:43:23Z")
Hello. In general, our application is writing one-line json log entity, but sometimes it`s multiline json, to merge multiline json i edited filebeat config as follows: ``` - input_type: log paths: - /var/log/application/eai-service.json json.message_key: message json.keys_under_root: true json.add_error_key: true json.overwrite_keys: true multiline: pattern: '^\{' negate: true match: after fields: tag: mp_eai fields_under_root: true ``` Example log: ``` {"@timestamp":"2018-07-18T17:15:02.082+04:00","@version":"1","message":"127.0.0.1 - - [2018-07-18T17:15:02.082+04:00] \"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0\" 200 21655","method":"GET","protocol":"HTTP/1.0","status_code":200,"requested_url":"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0","requested_uri":"/admin/tasks","remote_host":"127.0.0.1","content_length":21655,"elapsed_time":282} {"@timestamp":"2018-07-18T17:15:04.269+04:00","@version":"1","message":"127.0.0.1 - - [2018-07-18T17:15:04.269+04:00] \"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0\" 200 21655","method":"GET","protocol":"HTTP/1.0","status_code":200,"requested_url":"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0","requested_uri":"/admin/tasks","remote_host":"127.0.0.1","content_length":21655,"elapsed_time":92} ``` As result, this two log entities has been merged into one, in elasticsearch it looks like: ``` { "_index": "mp_eai-6.3.0-2018.29", "_type": "doc", "_id": "416HrWQBchTaZXe-Fi1s", "_version": 1, "_score": null, "_source": { "content_length": 21655, "tag": "mp_eai", "requested_uri": "/admin/tasks", "status_code": 200, "remote_host": "127.0.0.1", "protocol": "HTTP/1.0", "source": "/var/log/application/eai-service.json", "method": "GET", "host": { "name": "appserver" }, "beat": { "name": "server1", "hostname": "server1", "version": "6.3.0" }, "elapsed_time": 92, "@timestamp": "2018-07-18T13:15:04.269Z", "offset": 3330035, "tags": [ "beats_input_codec_plain_applied" ], "message": "127.0.0.1 - - [2018-07-18T17:15:02.082+04:00] \"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0\" 200 21655\n127.0.0.1 - - [2018-07-18T17:15:04.269+04:00] \"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0\" 200 21655", "@version": "1", "requested_url": "GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0" }, "fields": { "@timestamp": [ "2018-07-18T13:15:04.269Z" ] }, "sort": [ 1531919704269 ] } ``` Logstash output: ``` output { if [tag] == "mp_eai" { elasticsearch { hosts => ["elkserver1:9200", "elkserver2:9200", "elkserver3:9200"] user => logstash_internal password => testpassword sniffing => true manage_template => false index => "mp_eai-%{[@metadata][version]}-%{+xxxx.ww}" } } stdout { codec => rubydebug } } ``` Could you please advise, whats wrong? ---
**Author:** ![Andrew\_Cholakian1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_cholakian1/32/3612_2.png) [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)\ **Post date:** [July 18, 2018, 10:40pm UTC](https://discuss.elastic.co/t/filebeat-merged-2-normal-lines/140580/2 "2018-07-18T22:40:51Z")
I'm not sure why the `json` options are working, but using the [processor](https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html) does. For example: ```auto filebeat.inputs: - type: tcp host: "localhost:7070" multiline: pattern: '^\{' negate: true match: after fields: tag: mp_eai processors: - decode_json_fields: fields: ["message"] ``` and running `head multi.json | nc localhost 7070` where `multi.json` includes your sample data. ---
**Author:** ![Ilia\_Matveikin](https://avatars.discourse-cdn.com/v4/letter/i/ecae2f/32.png) [@Ilia\_Matveikin](https://discuss.elastic.co/u/Ilia_Matveikin)\ **Post date:** [July 19, 2018, 5:32am UTC](https://discuss.elastic.co/t/filebeat-merged-2-normal-lines/140580/3 "2018-07-19T05:32:49Z")
Hi Andrew, Thanks for reply. Actually, I`m wondering why the message fields have been merged. Btw, now I cant reproduce the issue. Could you please confirm that config I`ve shown will merge multiline, eg.: ``` { "@timestamp":"2018-07-18T17:15:04.269+04:00", "@version":"1", "message":"127.0.0.1 - - [2018-07-18T17:15:04.269+04:00] \"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0\" 200 21655","method":"GET","protocol":"HTTP/1.0", "status_code":200, "requested_url":"GET /admin/tasks?page=0&size=20&sort=created,desc HTTP/1.0", "requested_uri":"/admin/tasks", "remote_host": "127.0.0.1", "content_length":21655, "elapsed_time":92 } ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [July 19, 2018, 6:55am UTC](https://discuss.elastic.co/t/filebeat-merged-2-normal-lines/140580/4 "2018-07-19T06:55:15Z")
The reasons the processor works and json does is the order of the execution. Multiline is applied after the json decoder and is part of the reader / harvester. All the processors are only applied after the data has been collected by the harvester so they are applied after multiline. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2018, 6:55am UTC](https://discuss.elastic.co/t/filebeat-merged-2-normal-lines/140580/5 "2018-08-16T06:55:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.