# Logstash Simple JSON

**URL:** <https://discuss.elastic.co/t/logstash-simple-json/127351>\
**Category:** Logstash\
**Created:** [April 9, 2018, 3:42pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351 "2018-04-09T15:42:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\
**Post date:** [April 9, 2018, 3:42pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/1 "2018-04-09T15:42:34Z")

</div>

Trying to parse the following JSON file with Logstash:

> ```
> {
> "glossary":
> {
> "title": "example glossary",
> "GlossDiv":
> {
> "title": "S",
> "GlossList":
> {
> "GlossEntry":
> {
> "ID": "SGML",
> "SortAs": "SGML",
> "GlossTerm": "Standard Generalized Markup Language",
> "Acronym": "SGML",
> "Abbrev": "ISO 8879:1986",
> "GlossDef":
> {
> "para": "A meta-markup language, used to create markup languages such as DocBook.",
> "GlossSeeAlso": ["GML", "XML"]
> },
> "GlossSee": "markup"
> }
> }
> }
> }
> }
> 
> ```

With the following Logstash config file:

> ```
> input {
> file {
> type => "json"
> path => "/home/user/Elastic/logstash-6.2.3/data/queue/testJSON.json"
> start_position => "beginning"
> sincedb_path => "/dev/null"
> }
> }
> 
> filter {
> json {
> source => "glossary"
> }
> }
> 
> output {
> stdout { codec => rubydebug }
> elasticsearch{
> hosts => ["localhost:9200"]
> user => elastic
> password => changeme
> index => "test_index1"
> document_type => "json"
> }
> }
> 
> ```

Output in Kibana Dev Tools:

 ![Logstash](https://us1.discourse-cdn.com/elastic/original/3X/0/5/0502e8130ef5b056839d66f34d7e34a1370998c7.PNG)

Logstash isn't parsing it correctly. It is parsing the JSON line-by-line, but I want the entire JSON file to be one document. What am I doing wrong? I want each JSON file to be it's own event while maintaining the JSON's data structure in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2018, 4:04pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/2 "2018-04-09T16:04:34Z")

</div>

The use case of reading an entire file as a single event is something logstash does not handle well. You could do it using this, but you need to kill logstash after it reads the file.

```auto
input {
  file {
    path => "/.../test.json"
    codec => multiline {
      pattern => "^"
      negate => false
      what => "previous"
      auto_flush_interval => 2
    }
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter { json { source => "message" } }

```

---

<div class="post-metadata">

**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\
**Post date:** [April 9, 2018, 4:13pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/3 "2018-04-09T16:13:02Z")

</div>

@Badger What would be the best way to ingest multiple JSON files into Elasticsearch? (Where each JSON file is a single event)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2018, 4:46pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/4 "2018-04-09T16:46:28Z")

</div>

Starting a new instance of logstash for each file is really expensive. The above will process one file, and you could have a script to kill it. Or you could use a stdin input (provided the file is less than 16 KB), and that terminate logstash on EOF. To use a single instance of logstash you could may go through kafka or an http input.

But if what you really need to do is to ingest JSON file into Elasticsearch, I would bypass logstash and do it using /bin/sh and curl.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2018, 6:25pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/5 "2018-04-09T18:25:52Z")

</div>

I wanted to know how this would work with an http input. It is very simple to configure. On the logstash side I just have

```
input { http { host => "127.4.31.9" port => 4000 } }
output { stdout { codec => rubydebug } }

```

and I can send a directory full of files to it using

```
for F in /etc/logstash/t.httpInput/data/*.xml ; do
  echo Processing $F
  curl -H 'content-type: application/xml' -XPUT 'http://127.4.31.9:4000/' -d "@$F"
  echo ''
done

```

I though I was going to have to configure a multiline code, but no, each PUT results in a single event.

---

<div class="post-metadata">

**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\
**Post date:** [April 12, 2018, 1:59pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/6 "2018-04-12T13:59:06Z")

</div>

Fixed:

Used the following Logstash config file:

> ```
> input {
> file {
> type => "json"
> path => "/home/prism-dev/Elastic/logstash-6.2.3/data/queue/*.json"
> start_position => "beginning"
> sincedb_path => "/dev/null"
> codec => "json"
> }
> }
> 
> filter {
> json {
> source => "message"
> }
> }
> 
> output {
> stdout { codec => rubydebug }
> 
> elasticsearch{
> hosts => ["localhost:9200"]
> user => elastic
> password => changeme
> index => "testindex"
> document_type => "json"
> }
> }
> 
> ```

@Badger This works for reading multiple JSON files where each JSON file holds 1 event.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2018, 2:05pm UTC](https://discuss.elastic.co/t/logstash-simple-json/127351/7 "2018-05-10T14:05:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.