# Input JSON file to elasticsearch via logstash

**URL:** <https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437>\
**Category:** Logstash\
**Created:** [January 23, 2017, 6:44am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437 "2017-01-23T06:44:48Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 6:44am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/1 "2017-01-23T06:44:48Z")

</div>

hi, im facing an issue with logstash while inserting json data to elasticsearch.

here is my logstash config file

> input  
> {  
> file  
> {  
> codec =\> multiline  
> {  
> pattern =\> '^{'  
> negate =\> true  
> what =\> previous  
> }

> ```
> path => ["/usr/local/Cellar/logstash/5.1.1/test_payment.json"]
> start_position => "beginning"
> sincedb_path => "/dev/null"
> codec => "json"
> }
> 
> ```
> 
> }

> filter  
> {  
> mutate  
> {  
> replace =\> ["message", "%{message}"]  
> gsub =\> ['message','\n','']  
> }  
> if [message] =~ /^{.\*}$/  
> {  
> json { source =\> message }  
> }

> }

> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> user =\> elastic  
> password =\> techno2013  
> index =\> "test\_payment"  
> }  
> stdout { codec =\> rubydebug }  
> }

then, this is my json file

> {  
> "user\_id":0,  
> "cart\_id":"81746",  
> "error\_status":"",  
> "error\_message":"",  
> "data":{  
> "ord\_id":218308,  
> "cart":{  
> "cart\_id":"81746",  
> "session\_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",  
> "ip\_address":"180.254.65.91",  
> "usr\_id":"0",  
> "created\_date":"2017-01-19 17:29:20",  
> "updated\_date":"2017-01-19 18:08:35"  
> },  
> "cart\_data":{  
> "coupon\_code":"",  
> "[ord\_email":"shanti.swaragama@gmail.com](mailto:ord_email%22:%22shanti.swaragama@gmail.com)",  
> "ord\_firstname":"adhe",  
> "ord\_lastname":"shanti",  
> "ord\_address\_line1":"Radio Swaragama, Komplek Bulaksumur Blok H No 5",  
> "ord\_address\_line2":"",  
> "country\_id":"1",  
> "prov\_id":"9",  
> "city\_id":"45",  
> "ord\_postcode":"55281",  
> "ord\_phone":"082227369636",  
> "dif\_addr":null,  
> "rec\_name":"",  
> "rec\_address\_line1":"",  
> "rec\_address\_line2":"",  
> "rec\_country":"1",  
> "rec\_prov":"10",  
> "rec\_city":"0",  
> "rec\_postcode":"",  
> "rec\_phone":"",  
> "wrap\_type":"1",  
> "psncard\_type":null,  
> "psncard\_message":null,  
> "currency":"IDR",  
> "currency\_value":1,  
> "language":"en",  
> "is\_mobile":"0",  
> "ord\_shp\_fee":18000,  
> "ord\_shp\_mode":"reg",  
> "ord\_py\_mode":"transfer",  
> "newtoken":"",  
> "cc\_credit\_option":null,  
> "cc\_saved\_bank":"",  
> "cc\_saved\_period":"",  
> "cc\_savecc":null,  
> "cc\_saved\_no":null,  
> "cc\_savedccv":null,  
> "cc\_new\_bank":"",  
> "cc\_new\_period":"",  
> "cc\_newcc1":"",  
> "cc\_newcc2":"",  
> "cc\_newcc3":"",  
> "cc\_newcc4":"",  
> "cc\_newexp1":"",  
> "cc\_newexp2":"",  
> "cc\_newccv":""  
> },  
> "cart\_detail":[  
> {  
> "cadet\_id":"179977",  
> "cart\_id":"81746",  
> "stk\_id":"5240",  
> "cadet\_qty":"1",  
> "cadet\_data":"",  
> "pd\_id":"4890",  
> "pd\_name":"Denim Odette",  
> "pd\_shortdesc":"Cut-out Shoulders Top",  
> "pd\_price":"249000.00",  
> "pd\_img\_primary":"DenimOdette\_260916\_04.jpg",  
> "color\_id":"145",  
> "pd\_special":"0",  
> "pd\_sale":"0",  
> "pd\_weight":"0.15",  
> "color\_name":"Blue Jeans",  
> "color\_thumbnail":"Blue-Jeans.jpg",  
> "size\_id":"40",  
> "size\_name":"Size L",  
> "size\_shortname":"L",  
> "cat\_name":"Tops",  
> "cat\_id":"6",  
> "sub\_id":"16",  
> "sub\_name":"Short Sleeved",  
> "str\_coll\_id":"2"  
> }  
> ],  
> "order\_summary":{  
> "err\_status":false,  
> "err\_msg":"",  
> "subtotal":249000,  
> "promotion":0,  
> "wrapping":0,  
> "shipping":18000,  
> "total":267000,  
> "total\_weight":0.15,  
> "coupon\_code":"",  
> "coupon\_type":"",  
> "boledisc":0,  
> "arrboledisc":[

> ```
> ],
> "bolefreeship":249000,
> "boleinstallment":249000,
> "shipping_code":"reg"
> }
> 
> ```
> 
> }  
> }

when i try running logstash -f logstash.conf  
all data successfully insert to elasticsearch, but the problem is  
i only got the structure of json data, the value of its json not inserted to elasticsearch.

for example i have this json file  
{"foo":"foo value", "bar": "bar value"}  
{"hello":"world", "goodnight": "moon"}

when i insert to elasticsearch, i only got foo, and bar, im not get foo value and bar value.

the question is, how can i inserted object and value to elasticsearch.  
im confused.  
please help me.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 6:46am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/2 "2017-01-23T06:46:49Z")

</div>

Your file input configuration is incorrect. You can't specify two codecs (multiline and json). In this case you'll want to pick the multiline codec and then you can add a json filter to parse the resulting JSON string.

> when i insert to elasticsearch, i only got foo, and bar, im not get foo value and bar value.

Please copy/paste the resulting document so it's more clear exactly what you mean.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 7:00am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/3 "2017-01-23T07:00:04Z")

</div>

the result is same sir.  
either using json or multiline.

i only got the object data, not the value.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/a/ac590dd51727490408f49dd91885687c6682a5c5.png)

**what i want is like this :**  
"user\_id":021  
"order\_id": 988982  
"email\_address": "gutasaputra@gmail.com"

**instead of :**  
"order\_id": {  
"type": "text",  
"fields": {  
"keyword": {  
"type": "keyword",  
"ignore\_above": 256  
}  
}

"email\_address": {  
"type": "text",  
"fields": {  
"keyword": {  
"type": "keyword",  
"ignore\_above": 256  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 7:05am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/4 "2017-01-23T07:05:57Z")

</div>

It seems like you're looking at the mapping of the index, not the documents in the index.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 7:14am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/5 "2017-01-23T07:14:15Z")

</div>

ya, its not there.  
no value of myobject inserted to elasticsearch.

how can i insert all of my objects and values inside of my json file to elasticsearch via logstash ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 7:18am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/6 "2017-01-23T07:18:09Z")

</div>

Judging by what you posted earlier you're not looking at the documents that Logstash inserts. You are looking at the mappings of the index, i.e. the collection of fields found in the documents (basically). Please show the actual contents of an example document.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 7:20am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/7 "2017-01-23T07:20:24Z")

</div>

this is my json file that i want to insert to elasticsearch via logstash sir

{  
"user\_id":0,  
"cart\_id":"81746",  
"error\_status":"",  
"error\_message":"",  
"data":{  
"ord\_id":218308,  
"cart":{  
"cart\_id":"81746",  
"session\_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",  
"ip\_address":"180.254.65.91",  
"usr\_id":"0",  
"created\_date":"2017-01-19 17:29:20",  
"updated\_date":"2017-01-19 18:08:35"  
},  
"cart\_data":{  
"coupon\_code":"",  
"ord\_email":"shanti.swaragama@gmail.com",  
"ord\_firstname":"adhe",  
"ord\_lastname":"shanti",  
"ord\_address\_line1":"Radio Swaragama, Komplek Bulaksumur Blok H No 5",  
"ord\_address\_line2":"",  
"country\_id":"1",  
"prov\_id":"9",  
"city\_id":"45",  
"ord\_postcode":"55281",  
"ord\_phone":"082227369636",  
"dif\_addr":null,  
"rec\_name":"",  
"rec\_address\_line1":"",  
"rec\_address\_line2":"",  
"rec\_country":"1",  
"rec\_prov":"10",  
"rec\_city":"0",  
"rec\_postcode":"",  
"rec\_phone":"",  
"wrap\_type":"1",  
"psncard\_type":null,  
"psncard\_message":null,  
"currency":"IDR",  
"currency\_value":1,  
"language":"en",  
"is\_mobile":"0",  
"ord\_shp\_fee":18000,  
"ord\_shp\_mode":"reg",  
"ord\_py\_mode":"transfer",  
"newtoken":"",  
"cc\_credit\_option":null,  
"cc\_saved\_bank":"",  
"cc\_saved\_period":"",  
"cc\_savecc":null,  
"cc\_saved\_no":null,  
"cc\_savedccv":null,  
"cc\_new\_bank":"",  
"cc\_new\_period":"",  
"cc\_newcc1":"",  
"cc\_newcc2":"",  
"cc\_newcc3":"",  
"cc\_newcc4":"",  
"cc\_newexp1":"",  
"cc\_newexp2":"",  
"cc\_newccv":""  
},  
"cart\_detail":[  
{  
"cadet\_id":"179977",  
"cart\_id":"81746",  
"stk\_id":"5240",  
"cadet\_qty":"1",  
"cadet\_data":"[]",  
"pd\_id":"4890",  
"pd\_name":"Denim Odette",  
"pd\_shortdesc":"Cut-out Shoulders Top",  
"pd\_price":"249000.00",  
"pd\_img\_primary":"DenimOdette\_260916\_04.jpg",  
"color\_id":"145",  
"pd\_special":"0",  
"pd\_sale":"0",  
"pd\_weight":"0.15",  
"color\_name":"Blue Jeans",  
"color\_thumbnail":"Blue-Jeans.jpg",  
"size\_id":"40",  
"size\_name":"Size L",  
"size\_shortname":"L",  
"cat\_name":"Tops",  
"cat\_id":"6",  
"sub\_id":"16",  
"sub\_name":"Short Sleeved",  
"str\_coll\_id":"2"  
}  
],  
"order\_summary":{  
"err\_status":false,  
"err\_msg":"",  
"subtotal":249000,  
"promotion":0,  
"wrapping":0,  
"shipping":18000,  
"total":267000,  
"total\_weight":0.15,  
"coupon\_code":"",  
"coupon\_type":"",  
"boledisc":0,  
"arrboledisc":[

```
     ],
     "bolefreeship":249000,
     "boleinstallment":249000,
     "shipping_code":"reg"
  }

```

}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 7:30am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/8 "2017-01-23T07:30:11Z")

</div>

Yes, but what does that document look like _in Elasticsearch_?

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 7:34am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/9 "2017-01-23T07:34:08Z")

</div>

do yo mean this ?

 ![](https://us1.discourse-cdn.com/elastic/original/2X/e/eae09a66f93d910822ee1f3583c2db1ea54179f1.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 8:25am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/10 "2017-01-23T08:25:35Z")

</div>

No! Use Kibana or the search API to look at the documents.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 9:19am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/11 "2017-01-23T09:19:14Z")

</div>

thats from kibana sir.

here is the full screenshot.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/a/a02b44e5ef28fde7df43b75cbccaf88ca0dce025.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 9:26am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/12 "2017-01-23T09:26:18Z")

</div>

That REST endpoint doesn't return any documents. Use the Discover tab instead or `GET /test_payment/_search` if you feel you need to use raw REST calls.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 23, 2017, 9:43am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/13 "2017-01-23T09:43:15Z")

</div>

ohya, i got it.  
i can see the document right now.

but i cant access it via kibana .  
when i try to discover data using kibana, it return No results found.

oh should i create new threat for this issue?

thanks for help

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2017, 10:01am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/14 "2017-01-23T10:01:42Z")

</div>

Things to check:

- Make sure the time span chosen in the Discover tab is correct and encompasses the `@timestamp` field (which is in UTC).
- Make sure the index pattern used matches the test\_payment index where your data seems to be stored.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2017, 10:01am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437/15 "2017-02-20T10:01:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.