# Parsing array of json objects with logstash and injesting to elastic

**URL:** <https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197>\
**Category:** Logstash\
**Created:** [October 11, 2019, 9:16am UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197 "2019-10-11T09:16:31Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sidharth\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sidharth_sinha/32/46287_2.png) [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\
**Post date:** [October 11, 2019, 9:16am UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/1 "2019-10-11T09:16:31Z")

</div>

Hi,

I am trying to injest data from logstash to elastic, and have this array of json objects, such that each element in the array is a doc in elasticsearch, with the key name as the keys in the json.

```
    [
    {"name": "bouza", "age": 40, "type": "customer", "credit": "Nil", "date":"2019-10-08T22:52:31-07:00"},
    {"name": "carmen", "age": 20, "type": "customer", "credit": "Nil", "date":"2019-10-09T21:11:01-07:00"},
    {"name": "karen", "age": 31, "type": "customer", "credit": "Nil", "date":"2019-10-08T20:09:16-07:00"},
    {"name": "varmin", "age": 24, "type": "customer", "credit": "Nil", "date":"2019-10-08T12:21:45-07:00"},
    ]

```

I tried this, but logstash doesnt do anything when i run it:

```
input {

  file {
    path => "/home/waldo/credit_data/*.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => json_lines
}

    }
    output {
      elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "credit_data"
      }
    }

```

Tried both json and json lines. It doesnt seem to do anything. Also I would want the date to be filtered into elastic search as a date field.

I searched for this simple question, and wasnt able to find the answer although many of them have posted this and they have found the solution.

Any help is appreciated. Thankyou so much in advance!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 11, 2019, 4:03pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/2 "2019-10-11T16:03:37Z")

</div>

None of the individual lines are valid JSON. The entire array is almost valid JSON (you need to remove the , that precedes the ]). You can read the entire file as a single event using a multiline codec with a pattern that never matches

```
codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }

```

Then split the array

```
    split { field => "someField" }
    date { match => ["[someField][date]", "YYYY-MM-dd'T'HH:mm:ssZZ" ] }

```

If you need to move the contents of [someField] to the top level you can do it in a ruby filter similar to [this](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/2).

---

<div class="post-metadata">

**Author:** ![Sidharth\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sidharth_sinha/32/46287_2.png) [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\
**Post date:** [October 12, 2019, 10:43am UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/3 "2019-10-12T10:43:43Z")

</div>

Thanks a lot Badger!

Indeed my array actually does not have the "," in the end. It was a typo.

What does the pattern here mean? "^Spalanzani"?

> [@Badger](#):
>
> Spalanzani

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 12, 2019, 12:42pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/4 "2019-10-12T12:42:28Z")

</div>

"^Spalanzani" just means a line starting with the word Spalanzani. That never matches the actual contents of the file, so combined with negate =\> true it matches every line, so that the entire contents of the file are joined into one event.

---

<div class="post-metadata">

**Author:** ![Sidharth\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sidharth_sinha/32/46287_2.png) [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\
**Post date:** [October 14, 2019, 6:05am UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/5 "2019-10-14T06:05:39Z")

</div>

Thanks Badger. Got it!

I tried modifying my conf file:

```
input {

  file {
    path => "/home/waldo/credit_data/test.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline { 
	pattern => "^Spalanzani" 
	negate => true 
	what => previous 
	auto_flush_interval => 1 
	multiline_tag => "" 
	}
}

}

filter {
split { 
	field => "someField" 
      }

date { 
	match => ["[someField][date]", "YYYY-MM-dd'T'HH:mm:ssZZ" ] 
     }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "credit_data"
  }
}

```

Heres my json formatted data input:

```
[
  {
    "date": "2019-10-08T22:52:31-07:00",
    "credit": "Nil",
    "type": "customer",
    "age": 40,
    "name": "bouza"
  },
  {
    "date": "2019-10-09T21:11:01-07:00",
    "credit": "Nil",
    "type": "customer",
    "age": 20,
    "name": "carmen"
  },
  {
    "date": "2019-10-08T20:09:16-07:00",
    "credit": "Nil",
    "type": "customer",
    "age": 31,
    "name": "karen"
  },
  {
    "date": "2019-10-08T12:21:45-07:00",
    "credit": "Nil",
    "type": "customer",
    "age": 24,
    "name": "varmin"
  }
]

```

However, when I start logstash, I get:

```
[INFO] 2019-10-13 22:59:13.702 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[WARN] 2019-10-13 22:59:15.457 [[main]>worker16] split - Only String and Array types are splittable. field:someField is of type = NilClass

```

Do I need to define "someField" ? I was assuming that the key in the json should map to the elastic kv pair.

Thanks for your help so far!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 14, 2019, 12:03pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/6 "2019-10-14T12:03:33Z")

</div>

I forgot to mention the json filter

```
    filter { json { source => "message" target => "someField" remove_field => ["message"] } }
```

---

<div class="post-metadata">

**Author:** ![Sidharth\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sidharth_sinha/32/46287_2.png) [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\
**Post date:** [October 15, 2019, 5:18am UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/7 "2019-10-15T05:18:35Z")

</div>

> [@Badger](#):
>
> date { match =\> ["[someField][date]", "YYYY-MM-dd'T'HH:mm:ssZZ" ] }

Thanks Badger. Unfortunately, it doesnt seem to do anything.  
I added the below, but it doesnt ingest anything.

```
filter {

json {
         source => "message" target => "someField" remove_field => ["message"]
     }
split {
        field => "someField"
      }

date {
        match => ["[someField][date]", "YYYY-MM-dd'T'HH:mm:ssZZ" ]
     }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 15, 2019, 3:31pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/8 "2019-10-15T15:31:03Z")

</div>

I am unable to explain why that would not work.

---

<div class="post-metadata">

**Author:** ![Sidharth\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sidharth_sinha/32/46287_2.png) [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\
**Post date:** [October 15, 2019, 11:49pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/9 "2019-10-15T23:49:24Z")

</div>

Yeah. Thanks Badger.  
I ended up formatting my input to be individual json newline seperated events. And configured the filter accordingly. It worked for me.  
I will try to find out why the above doesnt work. THanks a lot for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 12, 2019, 11:49pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/10 "2019-11-12T23:49:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.