Accessing field of JSON Input

Elastic Stack Logstash
1

Hello!

I have a problem in accessing a json field. I've tried multiple solutions but had no success. What I've done:

I have this config file:

input {
  http {
    port => 5001
    codec => "json"
  }
}

filter {
  mutate {
    add_field => {"es_index" => "%{[statements][authority][name]}"}
  }
  mutate {
    gsub => [
      "es_index", " ", "_"
    ]
  }
  mutate {
    lowercase => ["es_index"]
  }
  ruby {
    init => "
        def remove_dots hash
            new = Hash.new
            hash.each { |k,v|
                if v.is_a? Hash
                    v = remove_dots(v)
                end
                new[ k.gsub('.','_') ] = v
                if v.is_a? Array
                    v.each { |elem|
                        if elem.is_a? Hash
                            elem = remove_dots(elem)
                        end
                        new[ k.gsub('.','_') ] = elem
                    } unless v.nil?
                end
            } unless hash.nil?
            return new
        end
    "
    code => "
        event.instance_variable_set(:@data,remove_dots(event.to_hash))
    "
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "golab-%{+YYYY.MM.dd}"
  }  
}

I have a filter with mutate. I want to add a field that I can use as a part of the index name. When I use this "%{[statements][authority][name]}" the content in the brackets is %{[statements][authority][name]}. Logstash seems to think this is a string?

I've also tried to use this expression: "%{statements}". It's working like expected. Everything in the field statements is passed to es_index. If I use "%{[statements][authority]}" strange things happen. es_index is filled with the exact same output that "%{statements}" produces. What am I missing?

Logstash Output with "%{[statements][authority]}" is:

{
    "statements" => {
             "verb" => {
                 "id" => "http://adlnet.gov/expapi/verbs/answered",
            "display" => {
                "en-US" => "answered"
            }
        },
          "version" => "1.0.1",
        "timestamp" => "2016-07-21T07:41:18.013880+00:00",
           "object" => {
            "definition" => {
                       "name" => {
                    "en-US" => "Example Activity"
                },
                "description" => {
                    "en-US" => "Example activity description"
                }
            },
                    "id" => "http://adlnet.gov/expapi/activities/example"
        },
            "actor" => {
               "account" => {
                "homePage" => "http://example.com",
                    "name" => "xapiguy"
            },
            "objectType" => "Agent"
        },
           "stored" => "2016-07-21T07:41:18.013880+00:00",
        "authority" => {
                  "mbox" => "mailto:info@golab.eu",
                  "name" => "GoLab",
            "objectType" => "Agent"
        },
               "id" => "0771b9bc-b1b8-4cb7-898e-93e8e5a9c550"
    },
            "id" => "a7e31874-780e-438a-874c-964373d219af",
      "@version" => "1",
    "@timestamp" => "2016-07-21T07:41:19.061Z",
          "host" => "172.23.0.3",
       "headers" => {
              "request_method" => "POST",
                "request_path" => "/",
                 "request_uri" => "/",
                "http_version" => "HTTP/1.1",
                   "http_host" => "logstasher:5001",
              "content_length" => "709",
        "http_accept_encoding" => "gzip, deflate",
                 "http_accept" => "*/*",
             "http_user_agent" => "python-requests/2.9.1",
             "http_connection" => "close",
                "content_type" => "application/json"
    },
      "es_index" => "{\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/answered\",\"display\":{\"en-us\":\"answered\"}},\"version\":\"1.0.1\",\"timestamp\":\"2016-07-21t07:41:18.013880+00:00\",\"object\":{\"definition\":{\"name\":{\"en-us\":\"example_activity\"},\"description\":{\"en-us\":\"example_activity_description\"}},\"id\":\"http://adlnet.gov/expapi/activities/example\",\"objecttype\":\"activity\"},\"actor\":{\"account\":{\"homepage\":\"http://example.com\",\"name\":\"xapiguy\"},\"objecttype\":\"agent\"},\"stored\":\"2016-07-21t07:41:18.013880+00:00\",\"authority\":{\"mbox\":\"mailto:info@golab.eu\",\"name\":\"golab\",\"objecttype\":\"agent\"},\"id\":\"0771b9bc-b1b8-4cb7-898e-93e8e5a9c550\"}"
}

You can see that authority is part of es_index. So it was not chosen as a field.

Many Thanks in advance!

2

What version of logstash are you using?

With 2.3.2, I had no issue setting the field. I removed the ruby code filter, and the codec => "json" for the http input (since it will do this automatically if the http header has "Content-Type: application/json". I also switched the field to use the logstash @metadata, since this would seem to be an ephemeral field that doesn't need to be stored in elasticsearch.

logstash config

input {
  http {
    port => 5001
  }
}

filter {
  if [statements] {
    mutate {
      add_field => {"[@metadata][es_index]" => "%{[statements][authority][name]}"}
    }
    mutate {
      gsub => [ "[@metadata][es_index]", " ", "_" ]
      lowercase => ["[@metadata][es_index]"]
    }
    mutate {
      add_field => {"[@metadata][es_index_full]" => "%{[@metadata][es_index]}-%{+YYYY.MM.dd}"}
    }
  }
}

output {
  # elasticsearch {
  #   hosts => "elasticsearch:9200"
  #   index => "golab-%{+YYYY.MM.dd}"
  # }
  stdout {
    codec => rubydebug { metadata => true }
  }
}

curl request

curl -XPOST -H "Content-Type: application/json" localhost:5001 -d '{
  "statements": {
    "verb": {
      "id": "http://adlnet.gov/expapi/verbs/answered",
      "display": {
        "en-US": "answered"
      }
    },
    "version": "1.0.1",
    "timestamp": "2016-07-21T07:41:18.013880+00:00",
    "object": {
      "definition": {
        "name": {
          "en-US": "Example Activity"
        },
        "description": {
          "en-US": "Example activity description"
        }
      },
      "id": "http://adlnet.gov/expapi/activities/example"
    },
    "actor": {
      "account": {
        "homePage": "http://example.com",
        "name": "xapiguy"
      },
      "objectType": "Agent"
    },
    "stored": "2016-07-21T07:41:18.013880+00:00",
    "authority": {
      "mbox": "mailto:info@golab.eu",
      "name": "GoLab",
      "objectType": "Agent"
    },
    "id": "0771b9bc-b1b8-4cb7-898e-93e8e5a9c550"
  }
}'

stdout

bin/logstash -f ../logstash-http-input.config
Settings: Default pipeline workers: 4
Pipeline main started
{
    "statements" => {
             "verb" => {
                 "id" => "http://adlnet.gov/expapi/verbs/answered",
            "display" => {
                "en-US" => "answered"
            }
        },
          "version" => "1.0.1",
        "timestamp" => "2016-07-21T07:41:18.013880+00:00",
           "object" => {
            "definition" => {
                       "name" => {
                    "en-US" => "Example Activity"
                },
                "description" => {
                    "en-US" => "Example activity description"
                }
            },
                    "id" => "http://adlnet.gov/expapi/activities/example"
        },
            "actor" => {
               "account" => {
                "homePage" => "http://example.com",
                    "name" => "xapiguy"
            },
            "objectType" => "Agent"
        },
           "stored" => "2016-07-21T07:41:18.013880+00:00",
        "authority" => {
                  "mbox" => "mailto:info@golab.eu",
                  "name" => "GoLab",
            "objectType" => "Agent"
        },
               "id" => "0771b9bc-b1b8-4cb7-898e-93e8e5a9c550"
    },
      "@version" => "1",
    "@timestamp" => "2016-07-23T16:38:36.422Z",
          "host" => "0:0:0:0:0:0:0:1",
       "headers" => {
         "request_method" => "POST",
           "request_path" => "/",
            "request_uri" => "/",
           "http_version" => "HTTP/1.1",
              "http_host" => "localhost:5001",
        "http_user_agent" => "curl/7.43.0",
            "http_accept" => "*/*",
           "content_type" => "application/json",
         "content_length" => "857"
    },
     "@metadata" => {
             "es_index" => "golab",
        "es_index_full" => "golab-2016.07.23"
    }
}
1 Like
3

Many Thanks for your solution! It now works. (elasticsearch 2.3.4)

It was some kind of a problem between keyboard and chair :wink:

I used your output code codec => rubydebug { metadata => true }
This showed me that statements (as the name already indicates) is an array...
So I had to use %{[statements][0][authority][name]} (I'm sure that only one element is in that array. If there are more, all of them have the same authority name =) )

I've used this array notation before, but had no success. Also I'm sure codec => rubydebug did not show me that statements is an array. Maybe it has something to do with removing the codec => 'json'?

Many thanks! Now I can work on :wink: