# JSON parsing question - Elasticsearch+Kibana+Logstash 6.5

**URL:** <https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255>\
**Category:** Logstash\
**Created:** [January 7, 2019, 6:29pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255 "2019-01-07T18:29:34Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![thiagotankian](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@thiagotankian](https://discuss.elastic.co/u/thiagotankian)\
**Post date:** [January 7, 2019, 6:29pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/1 "2019-01-07T18:29:35Z")

</div>

Hi,  
I'm a new user from Elastic and I'm trying to parse and index a json file using logstash, but so far I was able to do it but the fields were not indexed as expected - for example - all my fields were not indexed - I can only search the data as a String search.  
Could someone help me and let me know what I'm probably doing wrong?

Each object from my JSON file is a single line (my file has thousands of objects). See an example:

`{"_index":"test","_type":"log","_id":"jfdsjhadhjsdddshjkl","_score":1,"_source":{"@timestamp":"2019-01-07T02:02:21.567Z","beat":{"hostname":"ip-111-11-11-111","name":"ip-111-11-11-111","version":"1.0.0"},"field1":"test","field2":"test",..."fieldx":"test","fieldxy":{" __cachedRelations":{},"__ data":{"field1":"test","field2":"test",..."fieldn":"test"}," __persisted":false,"__ strict":false,"field1":"test"}],"source":"test","type":"log"}}`

My Logstash conf:

```
input {
 stdin {
  type => "stdin-type"
 }
 file {
  path => ["mypath/myfile.json"]
  sincedb_path => "nul"
  start_position => "beginning"
 }
}
filter {
}
}
output {
 elasticsearch {
   hosts => ["localhost:9200"]
 }
}

```

I also tried to insert a filter - json { source =\> "message" - but I started to face a lot of mapping issues (\_id, \_type and \_index metadata fields) that I could not find a way to solve.

Thanks

---

<div class="post-metadata">

**Author:** ![Chris\_Lyons](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_lyons/32/48107_2.png) [@Chris\_Lyons](https://discuss.elastic.co/u/Chris_Lyons)\
**Post date:** [January 7, 2019, 8:07pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/2 "2019-01-07T20:07:31Z")

</div>

Try adding the json\_lines codec to your input: [https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json\_lines.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html)

---

<div class="post-metadata">

**Author:** ![thiagotankian](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@thiagotankian](https://discuss.elastic.co/u/thiagotankian)\
**Post date:** [January 8, 2019, 12:35pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/3 "2019-01-08T12:35:58Z")

</div>

> [@thiagotankian](#):
>
> stdin { type =\> "stdin-type" }

No Luck Chris, my logstash starts with no errors but I can't see the index being created.

```
input {
 file {
  path => [mypath/myfiile.json"]
  sincedb_path => "null"
  start_position => "beginning"
  codec => "json_lines"
 }
}
filter {
}
output {
 elasticsearch {
   hosts => ["localhost:9200"]
 }
 stdout { 
  codec => rubydebug
 }
}

```

I also tried to apply new modifications after some additional research here in the discussion website, but now I am facing another issue:

> Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2019.01.08", :\_type=\>"doc", :routing=\>nil}, #LogStash::Event:0x39d97ce5], :response=\>{"index"=\>{"\_index"=\>"logstash-2019.01.08", "\_type"=\>"doc", "\_id"=\>"G4Z1LmgBAfcjVD\_FA9s7", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Field [\_source] is defined both as an object and a field in [doc]"}}}}

Here is my conf file:

```
input {
 file {
  path => ["mypath/myfile.json"]
  sincedb_path => "null"
  start_position => "beginning"
 }
}
filter {
 json {
  source => "message"
 }
 mutate {
  rename => { "_id" => "idoriginal" }
  rename => { "_index" => "indexoriginal" }
  rename => { "_type" => "typeoriginal" }
 }
}
output {
 elasticsearch {
   hosts => ["localhost:9200"]
 }
 stdout { 
  codec => rubydebug
 }
}

```

Any other ideas?

Thanks

---

<div class="post-metadata">

**Author:** ![Chris\_Lyons](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_lyons/32/48107_2.png) [@Chris\_Lyons](https://discuss.elastic.co/u/Chris_Lyons)\
**Post date:** [January 10, 2019, 4:43am UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/4 "2019-01-10T04:43:34Z")

</div>

I believe your JSON is not valid JSON. I took your example above and cleaned it up:

> {"\_index":"test","\_type":"log","\_id":"jfdsjhadhjsdddshjkl","\_score":1,"\_source":{"@timestamp":"2019-01-07T02:02:21.567Z","beat":{"hostname":"ip-111-11-11-111","name":"ip-111-11-11-111","version":"1.0.0"},"field1":"test","field2":"test","fieldx":"test","fieldxy":{"\_\_cachedRelations":{},"\_\_data":{"field1":"test","field2":"test","fieldn":"test"},"\_\_persisted":false,"\_\_strict":false,"field1":"test"},"source":"test","type":"log"}}

This is my LS config for testing:

> /usr/share/logstash/bin/logstash -e 'input{stdin{codec=\>"json\_lines"}}output{stdout{codec=\>rubydebug}}'

Output:

> {  
> "@version" =\> "1",  
> "\_type" =\> "log",  
> "\_source" =\> {  
> "source" =\> "test",  
> "beat" =\> {  
> "name" =\> "ip-111-11-11-111",  
> "version" =\> "1.0.0",  
> "hostname" =\> "ip-111-11-11-111"  
> },  
> "field2" =\> "test",  
> "fieldx" =\> "test",  
> "type" =\> "log",  
> "fieldxy" =\> {  
> "\_\_cachedRelations" =\> {},  
> "\_\_data" =\> {  
> "field1" =\> "test",  
> "field2" =\> "test",  
> "fieldn" =\> "test"  
> },  
> "field1" =\> "test",  
> "\_\_strict" =\> false,  
> "\_\_persisted" =\> false  
> },  
> "field1" =\> "test",  
> "@timestamp" =\> "2019-01-07T02:02:21.567Z"  
> },  
> "\_id" =\> "jfdsjhadhjsdddshjkl",  
> "host" =\> "",  
> "\_index" =\> "test",  
> "@timestamp" =\> 2019-01-10T04:33:46.976Z,  
> "\_score" =\> 1  
> }

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 10, 2019, 10:24pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/5 "2019-01-10T22:24:19Z")

</div>

> [@thiagotankian](#):
>
> "reason"=\>"Field [\_source] is defined both as an object and a field in [doc]"

Your JSON document includes a top-level field called \_source, which is an object. However, elasticsearch has [its own use](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html) for that field.

If you start off with an empty index does the error go away?

---

<div class="post-metadata">

**Author:** ![thiagotankian](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@thiagotankian](https://discuss.elastic.co/u/thiagotankian)\
**Post date:** [January 11, 2019, 12:12pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/6 "2019-01-11T12:12:12Z")

</div>

Thanks @Badger, but what do you mean with "start off with an empty index"?  
I not created and not mapped the fields before execute this conf file. The index could be created automatic, right? About the mapping, is there a way to make it dynamically? Because I now that I can have additional fields in some specific lines.

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 11, 2019, 1:04pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/7 "2019-01-11T13:04:14Z")

</div>

You appear to be using daily indexes. Does it happen for the first document that you insert on a new day (noting that es rolls daily indexes at midnight UTC).

---

<div class="post-metadata">

**Author:** ![Chris\_Lyons](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_lyons/32/48107_2.png) [@Chris\_Lyons](https://discuss.elastic.co/u/Chris_Lyons)\
**Post date:** [January 11, 2019, 3:48pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/8 "2019-01-11T15:48:46Z")

</div>

Badger is right and I over looked your full JSON content. You are using metadata fields in the JSON and you accounted for most with exception of \_source and apparently \_score just gets ignored. Why do you have those fields in your JSON? It's almost as if you pulled the document from elasticsearch.

I changed the field names in the JSON and document indexes into ES perfectly:

> {"my\_index":"test","my\_type":"log","my\_id":"jfdsjhadhjsdddshjkl","\_score":100,"my\_source":{"@timestamp":"2019-01-07T02:02:21.567Z","beat":{"hostname":"ip-111-11-11-111","name":"ip-111-11-11-111","version":"1.0.0"},"field1":"test","field2":"test","fieldx":"test","fieldxy":{"\_\_cachedRelations":{},"\_\_data":{"field1":"test","field2":"test","fieldn":"test"},"\_\_persisted":false,"\_\_strict":false,"field1":"test"},"source":"test","type":"log"}}

You can control the type, index, and document\_id via the ES output plugin if needed.

[Elasticsearch Metatdata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html)

This is the ES document created, checkout the metadata fields that are added:

> {  
> "\_index": "testing",  
> "\_type": "doc",  
> "\_id": "4GSSPWgBiOp7-GDzLdvq",  
> "\_version": 1,  
> "\_score": null,  
> "\_source": {  
> "@timestamp": "2019-01-11T15:40:16.633Z",  
> "my\_index": "test",  
> "my\_id": "jfdsjhadhjsdddshjkl",  
> "my\_source": {  
> "field1": "test",  
> "@timestamp": "2019-01-07T02:02:21.567Z",  
> "fieldx": "test",  
> "field2": "test",  
> "fieldxy": {  
> "\_\_persisted": false,  
> "field1": "test",  
> "\_\_strict": false,  
> "\_\_data": {  
> "field1": "test",  
> "field2": "test",  
> "fieldn": "test"  
> },  
> "\_\_cachedRelations": {}  
> },  
> "source": "test",  
> "beat": {  
> "hostname": "ip-111-11-11-111",  
> "version": "1.0.0",  
> "name": "ip-111-11-11-111"  
> },  
> "type": "log"  
> },  
> "my\_type": "log",  
> "\_score": 100,  
> "host": "",  
> "@version": "1"  
> },  
> "fields": {  
> "@timestamp": [  
> "2019-01-11T15:40:16.633Z"  
> ],  
> "my\_source.@timestamp": [  
> "2019-01-07T02:02:21.567Z"  
> ]  
> },  
> "sort": [  
> 1547221216633  
> ]  
> }

---

<div class="post-metadata">

**Author:** ![thiagotankian](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@thiagotankian](https://discuss.elastic.co/u/thiagotankian)\
**Post date:** [January 11, 2019, 4:03pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/9 "2019-01-11T16:03:54Z")

</div>

Thanks @Badger and @Chris_Lyons.

My file is an elasticsearch extraction from another server (using elasticdump), that is why I have to import to my server now.  
Unfortunately my file has around 100 million records and will be difficult to rename the "\_source" field. Is there a way to rename it in my logstash conf file?

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 11, 2019, 4:06pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/10 "2019-01-11T16:06:44Z")

</div>

Yes, you can use [mutate+rename](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename)

---

<div class="post-metadata">

**Author:** ![Chris\_Lyons](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_lyons/32/48107_2.png) [@Chris\_Lyons](https://discuss.elastic.co/u/Chris_Lyons)\
**Post date:** [January 11, 2019, 4:59pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/11 "2019-01-11T16:59:23Z")

</div>

You can give this a try too. You will need to install the prune filter:

> input {  
> file {  
> path =\> ["mypath/myfile.json"]  
> sincedb\_path =\> "null"  
> start\_position =\> "beginning"  
> codec =\> "json\_lines"  
> }  
> }  
> filter{  
> #Place your source into a new field which converts it back to a string  
> mutate{ add\_field=\>{"temp"=\>"%{\_source}"} }
> 
> #Leverage JSON filter to again parse the JSON and placing the contents of \_source at the root of the document  
> json{ source=\>"temp" }
> 
> #Run prune to clean things up  
> prune { blacklist\_names =\> ["^\_id$","^\_index$","^\_score$","^\_type$","^\_source$","^temp$"]}  
> }

[Working With Plugins](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html)

---

<div class="post-metadata">

**Author:** ![thiagotankian](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@thiagotankian](https://discuss.elastic.co/u/thiagotankian)\
**Post date:** [January 11, 2019, 7:13pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/12 "2019-01-11T19:13:00Z")

</div>

@Chris_Lyons @Badger  
I used "mutate+rename" and that worked.

Thanks a lot for your support and help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2019, 7:13pm UTC](https://discuss.elastic.co/t/json-parsing-question-elasticsearch-kibana-logstash-6-5/163255/13 "2019-02-08T19:13:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.