# How to store logs to json files that can be read in later

**URL:** <https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599>\
**Category:** Logstash\
**Created:** [June 12, 2018, 8:52pm UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599 "2018-06-12T20:52:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanketshinde](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\
**Post date:** [June 12, 2018, 8:52pm UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599/1 "2018-06-12T20:52:56Z")

</div>

I am sourcing some documents from a remote cluster and storing them in files on a local machine. I would like to store the documents as a valid json that I can later read in, say for example, in python.

I used the file plugin which has a default json-lines as the codec. This, however, doesn't seem to do the job as it saves the documents in a '\n' delimited format which is not a valid json.

Has anyone faced this issue before? Any leads on this?

The desired format is:

[  
doc1,  
doc2  
...  
]  
Thanks.

---

<div class="post-metadata">

**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\
**Post date:** [June 12, 2018, 9:26pm UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599/2 "2018-06-12T21:26:20Z")

</div>

This may be what you are wanting:

```auto
# bin/logstash-plugin install logstash-filter-uuid
filter {
   uuid {
     target => "uuid"
   }
}
output {
	file {
	 path => "/tmp/%{uuid}.json"
	 codec => "json"
	}
}

```

...however if you have a lot of files, I would not recommend this, you can take out your OS or file system trying to add millions of little files to a single directory. You may want to consider an intermediary system such as mysql or kafka (or even Elasticsearch), or maybe a batch them together in a single file with a codec such as avro or protobuf (or even json lines) that your later application (say python) can decode.

---

<div class="post-metadata">

**Author:** ![sanketshinde](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\
**Post date:** [June 12, 2018, 9:29pm UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599/3 "2018-06-12T21:29:43Z")

</div>

I edited the post, that should make clear what I seek to achieve. Thanks for the prompt reply!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 13, 2018, 6:14am UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599/4 "2018-06-13T06:14:48Z")

</div>

How would Logstash know when to write the final `]`?

Any program that's capable of reading

```nohighlight
[
{...},
{...}
...
]

```

would also be capable of reading this:

```nohighlight
{...}
{...}
...

```

Depending on the size of the file the latter might also be more efficient.

---

<div class="post-metadata">

**Author:** ![sanketshinde](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\
**Post date:** [June 14, 2018, 1:38pm UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599/5 "2018-06-14T13:38:51Z")

</div>

Sure, I just kept it that way. Thanks for the replies.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 12, 2018, 1:39pm UTC](https://discuss.elastic.co/t/how-to-store-logs-to-json-files-that-can-be-read-in-later/135599/6 "2018-07-12T13:39:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.