# Json file from filebeat to Logstash and then to elasticsearch **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 16, 2017, 11:25pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039 "2017-11-16T23:25:43Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 16, 2017, 11:25pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/1 "2017-11-16T23:25:43Z")
Hi I am few in setting filbeat, wondering i can get some advise . I am trying to ingested inventory data which is produced following json fileformat . { "\_meta": { "hostvars": { "host1": { "foreman": { "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": [ "build" ], "certname": "host1", "comment": "this is hostname1", "created\_at": "2017-03-08T15:27:11Z", "disk": "10gb", "domain\_id": 5, }, "foreman\_facts": { "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "1.1.1.1", "ipaddress\_eth0": "1.1.1.2", "ipaddress\_lo": "127.0.0.1", }, "foreman\_params": {} }, "host2": { "foreman": { "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": [ "build" ], "certname": "host2", "comment": "this hostname2", "created\_at": "2017-03-08T15:27:11Z", "disk": "20gb", "domain\_id": 5, }, "foreman\_facts": { "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "2.1.1.1", "ipaddress\_eth0": "2.2.2.2", "ipaddress\_lo": "127.0.0.1", }, "foreman\_params": {} }, "foreman\_all": [ "host3", "host4", ], "foreman\_environment: [ "computer1", "computer2" ], So only interested in **hostvars** and index the document based on the hostname and ignore **foreman\_all** and **foreman\_environment** fields . I want to send the json to Logstash where I want to further filter some of the json fields and rename some of the json fields and then send it to elastic search . I did open the topic in logstash section and they suggested to use filebeat multi line option to send the json data to logstash . I am using following filebeat option , however logstsh throw json error when i send the data from filebeat to logstash . filebeat.prospectors: - paths: - /var/log/mylog.json json.keys\_under\_root: true json.add\_error\_key: true **Final format in Elastic Search** Elastic doc id 1 computer name : "host1" "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": ["build"], "Company hardware name": "host1", "comment": "this is hostname1", "created\_at": "2017-03-08T15:27:11Z", "disk": "10gb", "domain\_id": 5, "foreman\_facts": { "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "1.1.1.1", "ipaddress\_eth0": "1.1.1.2", "ipaddress\_lo": "127.0.0.1", Elastic doc id 2 "computer name"" : "host2" "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": ["build"], "certname": "host2", "comment": "this hostname2", "created\_at": "2017-03-08T15:27:11Z", "disk": "20gb", "domain\_id": 5, "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "2.1.1.1", "ipaddress\_eth0": "2.2.2.2", "ipaddress\_lo": "127.0.0.1", ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 17, 2017, 12:12pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/2 "2017-11-17T12:12:19Z")
Wonder if I can get any advise on above please ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 17, 2017, 5:53pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/3 "2017-11-17T17:53:07Z")
Here is what i see in Elastic search even if I send the data directly from Filebeat . ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c668f5f7637ad4acc630ba5c1ed2fe7db61ab007.png) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 19, 2017, 11:00pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/4 "2017-11-19T23:00:25Z")
If you use Filebeat `json` feature, it expects the json on one line. You should use multiline in filebeat to make one event out of the json and then you can use logstash to decode the json and drop fields as needed. Please use code formatting with ticks when you paste code to make sure the indentations stays the same and makes it more readable. ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 20, 2017, 2:03pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/5 "2017-11-20T14:03:28Z")
Hi Ruflin, Thanks for the reply , I am using the following multiline config and for testing purposes i am sending data directly to elasticsearch . However it looks like filbeat unable to match the message . Is there any other setting I need to ingest the json data directly from filebeat to elasticsearch ### Multiline options multiline.pattern: ^{ multiline.negate: false ``` multiline.match: before ``` * * * ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/8/88032cb7ce20e1b8452e2afee16d22e8cded6f96.png) ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 20, 2017, 4:07pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/6 "2017-11-20T16:07:48Z")
Hi Manage to get the data in ELK using the following code . However I have noticed that filebeat treating whole json file as one message . Wondering If I can break the message and only send **hostvars** section and index the document based on the each **hostname** and ignore **foreman\_all** and **foreman\_environment** fields some thing like below Final format in Elastic Search Elastic doc id 1 computer name : "host1" "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": ["build"], "Company hardware name": "host1", "comment": "this is hostname1", "created\_at": "2017-03-08T15:27:11Z", "disk": "10gb", "domain\_id": 5, "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "1.1.1.1", "ipaddress\_eth0": "1.1.1.2", "ipaddress\_lo": "127.0.0.1", Elastic doc id 2 "computer name"" : "host2" "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": ["build"], "certname": "host2", "comment": "this hostname2", "created\_at": "2017-03-08T15:27:11Z", "disk": "20gb", "domain\_id": 5, "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "2.1.1.1", "ipaddress\_eth0": "2.2.2.2", "ipaddress\_lo": "127.0.0.1", **Following config worked :** multiline.pattern: '^{' multiline.negate: true multiline.match: after ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/4/343d07bee0d581966c2ab0617b6d55833567e142.png) ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 21, 2017, 12:07pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/7 "2017-11-21T12:07:40Z")
Hi Wondering if above is possible to break the multiline data at filebeat level. Regards Mussa shirazi ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 21, 2017, 11:09pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/8 "2017-11-21T23:09:51Z")
Glad you got it working with the pattern. To drop fields, you could use the drop\_fields processor: [https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html](https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html) If the hostname field always exists, you could use it as part of the index pattern. (see format string): [https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) As far as I understand after your multiline it is all still in one event. So you would need to use in addition the `decode_json` processor: [https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html](https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html) ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 23, 2017, 10:56am UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/9 "2017-11-23T10:56:01Z")
Hi Ruflin Thanks foe the last email , Unfortunately none of above work when i used the following setting . Still the message is ingested as one full message . which mean If I have 100 host name record all the data is sent as one message . Regarding the question about hostname fields , it always exists in the message but the names changes as need to ingest about 1k different host name used the fillowing config processors: - decode\_json\_fields: fields: ["message"] process\_array: false max\_depth: 1 overwrite\_keys: false processors: - drop\_fields: when: condition fields: ["\_meta","foreman\_all", foreman\_environment] * * * Still seen as one message in elastic search , I am using filebeat 5.6 , wondering if there are any improvements in version 6 fileabeat. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f6f23d2b585bd1217617e79dcd6ecc6aed101d56.png) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 26, 2017, 10:44pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/10 "2017-11-26T22:44:34Z")
Could you paste the full config with ticks around it to make sure you have the correct indentation? Also running with the `debug` log enabled could show you some more information. ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 27, 2017, 12:41pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/11 "2017-11-27T12:41:59Z")
Hi Ruflin, I have used the below configs as per suggestion if I use multiline option as mentioned below then I see the Filebeat and logstash send the whole json file as one message . This is what I looking to break the message based on host name as mentioned above . computer name : "host1" "architecture\_id": 1, "architecture\_name": "x86\_64", "capabilities": ["build"], "Company hardware name": "host1", "comment": "this is hostname1", "created\_at": "2017-03-08T15:27:11Z", "disk": "10gb", "domain\_id": 5, "foreman\_facts": { "boardmanufacturer": "Intel Corporation", "boardproductname": "440BX Desktop Reference Platform", "ipaddress": "1.1.1.1", "ipaddress\_eth0": "1.1.1.2", "ipaddress\_lo": "127.0.0.1", #=========================== Filebeat Configuration ============================= ``` filebeat.prospectors: - type: log # Change to true to enable this prospector configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /opt/uploaddata/*.json #- c:\programdata\elasticsearch\logs\* ### JSON configuration document_type: json json.message_key: log json.keys_under_root: true json.overwrite_keys: true #json.add_error_key: false multiline.pattern: '^{' multiline.negate: true multiline.match: after output.logstash: # The Logstash hosts hosts: ["localhost:5044"] #=========================== Logstash ============================= input { beats { port => "5044" } } filter { json { source => "parameter" target => "parameterData" remove_field => "parameter" } } output { elasticsearch { hosts => ["10.138.7.51:9200"] index => "inventory-%{+YYYY-MM-dd}" } stdout { codec => rubydebug } } #=========================== Filbear Errors ============================= 2017/11/24 16:45:14.226665 json.go:32: ERR Error decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {} 2017/11/24 16:45:14.226757 processor.go:262: DBG Publish event: { "@timestamp": "2017-11-24T16:45:14.226Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "6.0.0" }, "json": {}, "message": " \"host4\",", "prospector": { "type": "log" }, "beat": { "name": "filebeat", "hostname": "filebeat", "version": "6.0.0" }, "source": "/opt/uploaddata/data.json", "offset": 1710 } 2017/11/24 16:45:14.226800 json.go:32: ERR Error decoding JSON: EOF 2017/11/24 16:45:14.226889 processor.go:262: DBG Publish event: { "@timestamp": "2017-11-24T16:45:14.226Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "6.0.0" }, "json": {}, "message": "", "source": "/opt/uploaddata/data.json", "offset": 1712, "prospector": { "type": "log" }, "beat": { "name": "filebeat", "hostname": "filebeat", "version": "6.0.0" } #=========================== Logstash Logs ============================= { "@timestamp" => 2017-11-24T16:45:14.226Z, "offset" => 1638, "@version" => "1", "beat" => { "name" => "filebeat", "hostname" => "filebeat", "version" => "6.0.0" }, "host" => "filebeat", "prospector" => { "type" => "log" }, "json" => {}, "source" => "/opt/uploaddata/data.json", "message" => " },", "tags" => [ [0] "beats_input_codec_plain_applied" ] } { "@timestamp" => 2017-11-24T16:45:14.226Z, "offset" => 1666, "@version" => "1", "beat" => { "name" => "filebeat", "hostname" => "filebeat", "version" => "6.0.0" }, "host" => "filebeat", "json" => {}, "prospector" => { "type" => "log" }, "source" => "/opt/uploaddata/data.json", "message" => " \"foreman_all\":[ ", "tags" => [ [0] "beats_input_codec_plain_applied" ] } ``` ---
**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Post date:** [November 28, 2017, 6:22pm UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/12 "2017-11-28T18:22:59Z")
Hi Ruflin, Wondering you had a chance to look at the logs please . Regards Mussa shirazi ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 29, 2017, 1:57am UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/13 "2017-11-29T01:57:17Z")
In the config you shared about you use the `json.*` config options with `multiline`. Besides that the multiline indentation is wrong, you should use `multiline` with the `decode_json` processor and nothing the LS side or use `multiline` with the json filter in LS. Most important part first is fixing indentation of your multiline. I would advice you to turn of the json parts at first and make sure you get one json "line" per event after fixing the multiline. And then you can do the next step to decode the json. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 27, 2017, 1:57am UTC](https://discuss.elastic.co/t/json-file-from-filebeat-to-logstash-and-then-to-elasticsearch/108039/14 "2017-12-27T01:57:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.