# Simple example wanted: read and output json files to elasticsearch

**URL:** <https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410>\
**Category:** Logstash\
**Created:** [January 13, 2018, 3:35pm UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410 "2018-01-13T15:35:42Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jali](https://avatars.discourse-cdn.com/v4/letter/j/ecae2f/32.png) [@jali](https://discuss.elastic.co/u/jali)\
**Post date:** [January 13, 2018, 3:35pm UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410/1 "2018-01-13T15:35:42Z")

</div>

Hi! Basic stuff but cannot get this to work.  
I have a directory which contains bunch of json files (and new ones arriving every 10mins). The json files need to be read from dir and outputted to elastic.  
I want to read each json file and ouput as a new document in elastic. E.g. file 1234567.json would end up in elastic index "myindex" with document id "1234567". (The doc content should be the whole json from file). And I want to use samplePollTime as timefield in es/kibana.

Logstash current conf:  
input {  
file {  
path =\> "/data-IN/\*"  
sincedb\_path =\> "/dev/null"  
ignore\_older =\> 0  
}  
}

output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> ["localhost:9200"]  
document\_id =\> "%{doc\_id}"  
index =\> "myindex"  
}  
}

But stdout codec however show the json files processed as line by line? Broken into smaller objects with timestamp added into every contentN ?  
{  
"host" =\> "myhost",  
"path" =\> "data-IN",  
"@timestamp" =\> 2018-01-13T15:28:18.306Z,  
"@version" =\> "1",  
"message" =\> " "content1": {"  
}

So it is breaking the content into smaller objects.

Files are named like epochtimestamp.json and contain:  
{  
"content1": {  
"value": 29.697  
},  
"content2": {  
"ip": "10.10.10.10"  
},  
"content3": {  
"host": "somehost.domain.ex"  
},  
"samplePollTime": "2018-01-13T14:58:05.352Z",  
"notes": "more notes here"  
}

---

<div class="post-metadata">

**Author:** ![jali](https://avatars.discourse-cdn.com/v4/letter/j/ecae2f/32.png) [@jali](https://discuss.elastic.co/u/jali)\
**Post date:** [January 13, 2018, 6:10pm UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410/2 "2018-01-13T18:10:30Z")

</div>

Basically what I am saying is that Logstash seems to create a timestamped event from each object.key. And I don't want that. I need to have single document with timefield of samplePollTime.  
(Doing badly describing this, hopefully you understand it :-))

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2018, 9:16pm UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410/3 "2018-01-23T21:16:43Z")

</div>

The file input is normally line-oriented, so if you want to slurp the whole file at once you need to use a multiline codec. Examples of this has been posted in this past. Then, use a json filter to parse the JSON payload.

---

<div class="post-metadata">

**Author:** ![jali](https://avatars.discourse-cdn.com/v4/letter/j/ecae2f/32.png) [@jali](https://discuss.elastic.co/u/jali)\
**Post date:** [January 24, 2018, 9:47am UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410/4 "2018-01-24T09:47:10Z")

</div>

Ok, thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2018, 9:47am UTC](https://discuss.elastic.co/t/simple-example-wanted-read-and-output-json-files-to-elasticsearch/115410/5 "2018-02-21T09:47:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.