# Extract all jsons from log to one field

**URL:** <https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286>\
**Category:** Logstash\
**Created:** [January 16, 2020, 9:48am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286 "2020-01-16T09:48:23Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![dardev](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@dardev](https://discuss.elastic.co/u/dardev)\
**Post date:** [January 16, 2020, 9:48am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/1 "2020-01-16T09:48:23Z")

</div>

Hello, I'm trying to extract few possible jsons from log and put them to one field into an array.  
This is the example of log I want to parse:

> This is one of example logs: [{"lat":12.33,"lng":55.44}] that I should correctly parse [{"lat":12.33,"lng":55.44}]. It can contain multiple jsons [{"lat":12.33,"lng":55.44}].

The acceptance criteria is an field visible in kibana which will contain all these jsons put in one field for example "extracted\_objects": [{..}, {..}, {..}].  
I'm able to parse the first json using following config:

```
grok {
    match => { "message" => "%{CISCO_REASON}: \[%{GREEDYDATA:java_object_json}\].?" }
}

json {
    source => "java_object_json"
    target => "java_object"
}

```

but I hope this not the way I should follow... Because these jsons can be nested, it's hard to solve it using regexp.

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 16, 2020, 11:12am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/2 "2020-01-16T11:12:38Z")

</div>

> [@dardev](#):
>
> [{"lat":12.33,"lng":55.44}]

Does this solve your issue?

```
ruby {
  code => "
    require 'json'
    new_field = JSON.parse(event.get('message'))
    event.set('new_field', new_field)
  "
}

```

This assumes your input message is like `[{"lat":12.33,"lng":55.44}]`.

If that array is a part of your message (like `whatever you want before [{"lat":12.33,"lng":55.44}] whatever you want after`) first grok that part out in a field and then pass that field to the `JSON.parse ruby function` .

With this syntax, a pipeline like

```
input {
  stdin{}
}

filter {
  ruby {
    code => "
      require 'json'
      new_field = JSON.parse(event.get('message'))
      event.set('new_field', new_field)
    "
  }
}

output {
  stdout{}
}

```

Would act like this:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4cf45ec50276d3b3fd5eec53daaad9faa3acb948.png)

Is this what you're looking for?

---

<div class="post-metadata">

**Author:** ![dardev](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@dardev](https://discuss.elastic.co/u/dardev)\
**Post date:** [January 16, 2020, 11:29am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/3 "2020-01-16T11:29:57Z")

</div>

I can't determine how the log will look like. Log can be like:

> Test logging java object: [{"lat":12.33,"lng":55.44}] hjklgfdjhkl [{"lat":12.33,"lng":55.44}]

It just can contain multiple jsons inside plain text.

and your config will fail parsing that. What I really need is to have ability to look for a json pattern in the log and capture all matches. And then put all these matches into one field (array).

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 16, 2020, 11:50am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/4 "2020-01-16T11:50:12Z")

</div>

> [@dardev](#):
>
> [{"lat":12.33,"lng":55.44}]

Well, my first thought would be to tell whoever is generating this logs to fix them upstream since they cannot be that random.

Anyway, are the json object ALWAYS surrounded by square brackets? Is there any chance curly brackets are present in the log without them surrounding a json object?  
I mean, is somthing like the following a possible log?

`{"no_square":"valid_json"} qsdfwerg {invalid-->json} [{"valid":"json"}] qsfdw`

---

<div class="post-metadata">

**Author:** ![dardev](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@dardev](https://discuss.elastic.co/u/dardev)\
**Post date:** [January 16, 2020, 12:04pm UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/5 "2020-01-16T12:04:17Z")

</div>

Square brackets are not required. It's just by convention. It could be changed to \< \> or anything else if that will help.

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 16, 2020, 12:19pm UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/6 "2020-01-16T12:19:08Z")

</div>

No, I meant, all those json objects are always surrounded by square brackets (or anything else you choose)? Or is there any chance that a json object is outside the brackets?

---

<div class="post-metadata">

**Author:** ![dardev](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@dardev](https://discuss.elastic.co/u/dardev)\
**Post date:** [January 16, 2020, 12:41pm UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/7 "2020-01-16T12:41:21Z")

</div>

There is no chance. Printing an object (json) should be wrapped inside some brackets to make it more visible. And I can define something other than square brackets, because now it makes every json to be an array.

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 16, 2020, 1:06pm UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/8 "2020-01-16T13:06:34Z")

</div>

Ok so, IF the json objects in input are ALWAYS inside square brackets, this should do what you need:

```
input {
  stdin{}
}

filter {
  ruby {
    code => "
      require 'json'
      def valid_json?(json)
        JSON.parse(json)
          return true
        rescue JSON::ParserError => e
          return false
      end
      possible_jsons = event.get('message').scan(/(?<=\[).+?(?=\])/)
      jsons = possible_jsons.map{|item| JSON.parse(item) if valid_json?(item)}.compact
      event.set('new_field', jsons)
    "
  }
}

output {
  stdout{}
}

```

With this pipeline, here's the input-output  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/0/400a1070a958dfb888f24e22ff7a62d58430d50e.png)

---

<div class="post-metadata">

**Author:** ![dardev](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@dardev](https://discuss.elastic.co/u/dardev)\
**Post date:** [January 17, 2020, 6:23am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/9 "2020-01-17T06:23:58Z")

</div>

It's fine but works only for jsons without arrays. I changed surrounding square brackets to \< \> and now it works correctly even with nested jsons with arrays. So your code works great, I need only to find or create better regexp to handle surrounding square brackets. Thank you for help.

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 17, 2020, 9:29am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/10 "2020-01-17T09:29:27Z")

</div>

No problem!

Should you need help with the regex just ask.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2020, 9:29am UTC](https://discuss.elastic.co/t/extract-all-jsons-from-log-to-one-field/215286/11 "2020-02-14T09:29:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.