# Input Json file

**URL:** <https://discuss.elastic.co/t/input-json-file/107781>\
**Category:** Logstash\
**Created:** [November 15, 2017, 3:58pm UTC](https://discuss.elastic.co/t/input-json-file/107781 "2017-11-15T15:58:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\
**Post date:** [November 15, 2017, 3:58pm UTC](https://discuss.elastic.co/t/input-json-file/107781/1 "2017-11-15T15:58:03Z")

</div>

Hi

I am trying to ingest some of inventory data from JSON files using the following logstash config.

input {  
file {  
type =\> "json"  
path =\> "/opt/uploaddata/\*.json"  
start\_position =\> "beginning"  
}  
}  
filter {  
json {  
source =\> "message"  
target =\> "message"  
}  
}  
output {  
elasticsearch {  
hosts =\> ["10.138.7.51:9200"]  
index =\> "inventory-%{+YYYY-MM-dd}"  
}  
stdout {}  
}

**Format of Json file as follow .**

{  
"\_meta": {  
" **hostvars**": {  
" **host1**": {  
" **foreman**": {  
"architecture\_id": 1,  
"architecture\_name": "x86\_64",  
"capabilities": [  
"build"  
],  
"certname": "host1",  
"comment": "this is hostname1",  
"created\_at": "2017-03-08T15:27:11Z",  
"disk": "10gb",  
"domain\_id": 5,  
},  
" **foreman\_facts**": {  
"boardmanufacturer": "Intel Corporation",  
"boardproductname": "440BX Desktop Reference Platform",  
"ipaddress": "1.1.1.1",  
"ipaddress\_eth0": "1.1.1.2",  
"ipaddress\_lo": "127.0.0.1",  
},  
" **foreman\_params**": {}  
},  
" **host2**": {  
" **foreman**": {  
"architecture\_id": 1,  
"architecture\_name": "x86\_64",  
"capabilities": [  
"build"  
],  
"certname": "host2",  
"comment": "this hostname2",  
"created\_at": "2017-03-08T15:27:11Z",  
"disk": "20gb",  
"domain\_id": 5,  
},  
" **foreman\_facts**": {  
"boardmanufacturer": "Intel Corporation",  
"boardproductname": "440BX Desktop Reference Platform",  
"ipaddress": "2.1.1.1",  
"ipaddress\_eth0": "2.2.2.2",  
"ipaddress\_lo": "127.0.0.1",  
},  
"foreman\_params": {}  
},  
" **all**": [  
"host3",  
"host4",  
],  
**"foreman\_environment:** [  
"computer1",  
"computer2"  
],  
So only interested in **hostvars** and index the document based on the hostname and ingest is as follow and ignore **all** and **foreman\_environment**

Elastic doc id 1

computer name : "host1"  
"architecture\_id": 1,  
"architecture\_name": "x86\_64",  
"capabilities": ["build"],  
"certname": "host1",  
"comment": "this is hostname1",  
"created\_at": "2017-03-08T15:27:11Z",  
"disk": "10gb",  
"domain\_id": 5,  
"foreman\_facts": {  
"boardmanufacturer": "Intel Corporation",  
"boardproductname": "440BX Desktop Reference Platform",  
"ipaddress": "1.1.1.1",  
"ipaddress\_eth0": "1.1.1.2",  
"ipaddress\_lo": "127.0.0.1",

Elastic doc id 2

```
"computer name"" : "host2"
"architecture_id": 1, 
"architecture_name": "x86_64", 
"capabilities": ["build"], 
"certname": "host2", 
"comment": "this hostname2", 
"created_at": "2017-03-08T15:27:11Z", 
"disk": "20gb", 
"domain_id": 5, 
"boardmanufacturer": "Intel Corporation", 
"boardproductname": "440BX Desktop Reference Platform", 
"ipaddress": "2.1.1.1", 
"ipaddress_eth0": "2.2.2.2", 
"ipaddress_lo": "127.0.0.1", 

```

However I am getting following error when ingest the data using the above config and also elasticsearch add every json field as separate document .

**LogStash errors:**

at [Source: [B@45247ee5; line: 2, column: 3]\>}  
15:27:46.641 [[main]\>worker0] WARN logstash.filters.json - Error parsing json {:source=\>"message", :raw=\>" "\_meta": {\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: [B@49d56f1d; line: 1, column: 11]\>}  
15:27:46.641 [[main]\>worker0] WARN logstash.filters.json - Error parsing json {:source=\>"message", :raw=\>" "hostvars": {\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: [B@7a32a610; line: 1, column: 16]\>}  
15:27:46.642 [[main]\>worker0] WARN logstash.filters.json - Error parsing json {:source=\>"message", :raw=\>" "host1": {\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: [B@5137768b; line: 1, column: 15]\>}  
15:27:46.642 [[main]\>worker0] WARN logstash.filters.json - Error parsing json {:source=\>"message", :raw=\>" "foreman": {\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: [B@685351f2; line: 1, column: 19]\>}  
15:27:46.643 [[main]\>worker0] WARN logstash.filters.json - Error parsing json {:source=\>"message", :raw=\>" "architecture\_id": 1, \r", :exception=\>#\<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

**Elastic document :**

![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9c1ff7e92bc7627bb248bd52ca05662611730b23.png)

Please help in solving the above issues , I have also tried using **json\_lines** codec , however logstash does not even ingest single message with this Codec and dont see any activity on logstash window .

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 15, 2017, 4:19pm UTC](https://discuss.elastic.co/t/input-json-file/107781/2 "2017-11-15T16:19:37Z")

</div>

As I wrote in another topic yesterday: The file input reads files line by line. If you want to slurp an entire file into a single event you need to use a multiline codec. I'm sure examples have been posted in the past.

---

<div class="post-metadata">

**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\
**Post date:** [November 15, 2017, 4:30pm UTC](https://discuss.elastic.co/t/input-json-file/107781/3 "2017-11-15T16:30:30Z")

</div>

thanks Magnus ,

Wondering if you have link for the Topic you write yesterday

---

<div class="post-metadata">

**Author:** ![mussa572](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@mussa572](https://discuss.elastic.co/u/mussa572)\
**Post date:** [November 15, 2017, 4:36pm UTC](https://discuss.elastic.co/t/input-json-file/107781/4 "2017-11-15T16:36:36Z")

</div>

Also want to confirm what if we ingest the same json data from Filebeat , do we still need to use the muiltiline codec . As my final solution is to get the json file ingested via filebeat and send to logstash to make some changes on the format before sending it to elasticsearch

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 15, 2017, 5:01pm UTC](https://discuss.elastic.co/t/input-json-file/107781/5 "2017-11-15T17:01:27Z")

</div>

Generally, you will have a tough time with multiline or pretty-printed JSON in LS, this is because it is hard to know when the last `}` is reached especially if there is no newline after the `}` character.

When you use the `pattern => "^}$"` (a brace at the start of a line ) with `what => "previous"` the text satisfying the pattern is not seen as belonging to the previous and the accumulation of JSON lines has no final `}`.

Having said that a `pattern => "^{$"` with what next relies on a timeout flush feature to flush the accumulation buffer because another `\n{` is never seen. Filebeat's implementation of this timeout is better than the LS one.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 15, 2017, 6:21pm UTC](https://discuss.elastic.co/t/input-json-file/107781/6 "2017-11-15T18:21:50Z")

</div>

> Also want to confirm what if we ingest the same json data from Filebeat , do we still need to use the muiltiline codec

You need to use Filebeat's multiline feature. Not the multiline codec on the Logstash side.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2017, 6:21pm UTC](https://discuss.elastic.co/t/input-json-file/107781/7 "2017-12-13T18:21:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.