# How to read JSON files and insert in Elastic Stack

**URL:** <https://discuss.elastic.co/t/how-to-read-json-files-and-insert-in-elastic-stack/241229>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [July 15, 2020, 6:38am UTC](https://discuss.elastic.co/t/how-to-read-json-files-and-insert-in-elastic-stack/241229 "2020-07-15T06:38:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mike0211](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@Mike0211](https://discuss.elastic.co/u/Mike0211)\
**Post date:** [July 15, 2020, 6:38am UTC](https://discuss.elastic.co/t/how-to-read-json-files-and-insert-in-elastic-stack/241229/1 "2020-07-15T06:38:33Z")

</div>

I am new to Elastic. I have 10 independent docker containers each writing custom JSON files in a single directory. Each docker container writes file in a different format. Each file is a single line JSON along with timestamp. The files are getting written at the rate of 10 JSON files per docker container per second.

I want to:

1. Read the JSON files and insert into Elastic.
2. Delete the files that have been read.
3. Files should be read in first come first serve (though not a mandatory requirement but good to have)
4. Each JSON file type has to inserted into seperate index (i am thinking of having 10 indexes)
5. I am open to use Filebeats (but I did not get any option to delete the files) or logstash.

Any help in this regards is highly appreciated.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 15, 2020, 7:19am UTC](https://discuss.elastic.co/t/how-to-read-json-files-and-insert-in-elastic-stack/241229/2 "2020-07-15T07:19:07Z")

</div>

Welcome to our community! 😃

You can use Filebeat to ship the data to Elasticsearch. There's nothing in the stack that will delete the data though, you will need to manage that yourself.

---

<div class="post-metadata">

**Author:** ![Mike0211](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@Mike0211](https://discuss.elastic.co/u/Mike0211)\
**Post date:** [July 15, 2020, 7:48pm UTC](https://discuss.elastic.co/t/how-to-read-json-files-and-insert-in-elastic-stack/241229/3 "2020-07-15T19:48:48Z")

</div>

How do I know what files the Filebeat has processed. Maybe I can write a script to delete files but I need to know what has been processed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2020, 7:48pm UTC](https://discuss.elastic.co/t/how-to-read-json-files-and-insert-in-elastic-stack/241229/4 "2020-08-12T19:48:57Z")

</div>

Closed on August 12, 2020, 7:48pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.