Flatten json array in Logstash

Elastic Stack Logstash
1

Is there a way to flatten a json array in Logstash? Use-case is twitter input plugin, wanting to make array data usable in Kibana, c.f. Show images from URLs in data

thanks.

2

I don't understand what your messages look like. In the screenshot in your other thread the only array I see is indices: [139, 140].

3

@magnusbaeck In the tweet event, there's an array { "entities": { "media": [ .... ] } } that contains a list of objects (shown in the screenshot)

@rmoff In terms of making this visible in Kibana, I don't know. You could try using the ruby filter to make an array of whatever property you want:

Something like this may help you (I haven't tested it):

filter {
  ruby {
    code => 'event["images"] = event["[entities][media]"].collect { |m| m["media_url"] }
  }
}

The above should create a field "images" with value as an array of all the media_url values.

4

Thanks @jordansissel - good to know at least I'm not missing some obvious functionality :smile:
I tried your snippet (with the missing closing single quote added) with my existing simple config:

input {
    twitter {
        consumer_key => "foo"
        consumer_secret => "foo"
        oauth_token => "foo-foo"
        oauth_token_secret => "foo"
        keywords => ["foo","bar"]
        full_tweet => true
    }
}

filter {
          ruby {
            code => 'event["images"] = event["[entities][media]"].collect { |m| m["media_url"] }'
          }
}

output {
        elasticsearch { host => "localhost"  protocol => "http" index => "twitter"}
        stdout { codec => rubydebug }
        }

but got an error

Exception in filterworker {"exception"=>#<NoMethodError: undefined method `collect' for nil:NilClass>,

I'm no Ruby coder so I guess I'll have to leave this one here for now.

thanks again. Robin.

5

This should work:

code => 'event["images"] = event["[entities][media]"].collect { |m| m["media_url"] } unless event["[entities][media]"].nil?'
6

Top stuff -- worked a treat. Thanks so much (and for logstash in general).

7

I just came across this post again via a Google search when looking to solve exactly the same issue. On the basis that others might too, I wanted to add to it some ruby code that might be useful for others doing this, that will combine the elements of an array into a single string:

ruby {
code => 'event["hashtags_array"] = event["[entities][hashtags]"].collect { |m| m["text"] } unless event["[entities][hashtags]"].nil?
event["hashtags_list"] = event["hashtags_array"].join(",")  unless event["[hashtags_array]"].nil?'
}

The first line is just as @jordansissel provided me above; the second line is a hack together that I figured out but seems to work. Example input:

  "entities": {
    "hashtags": [
      {
        "text": "Iran",
        "indices": [
          18,
          23
        ]
      },
      {
        "text": "News",
        "indices": [
          24,
          29
        ]
      }
    ],

Example output:

          "hashtags_array" => [
    [0] "Iran",
    [1] "News"
],
            "hashtags_list" => "Iran,News",
8

I am worried this comment may be a little off topic, but thought I'd chime in here because I had a bit of trouble figuring out how to flatten an array in logstash 5. This post was the most relevant that I found that contained the crux of what I was trying to do.

Since the changed Ruby syntax in logstash 5 Event API, I've had to massage the join solution. Of course, I'm just squashing a basic array and don't need to collect the items from a JSON array in the original post. I just needed to check to see if the field is an array, and if so, join the elements together. Kind of fumbling my way through Ruby.

ruby => {
  code => 'if(!event.get("[logstash-array][1]").nil?)
    event.set("[logstash-string]", event.get("[logstash-array]").join(","))
    end'
}

Maybe it would be worth specifying how to flatten a JSON array with the limited Ruby support in the Event API as well for anyone who needs to do this in version 5. Since the hash syntax is no longer supported, it could be useful for people trying to find this solution moving forward.