How to Parse Json log format by ConfigMap in Logstash

[aleksei.saiko]

Hi,
The logs that I'm receiving for message, looking like that in Kibana -

message	       	{"time_date": "2019-02-14T14:00:39+00:00","client": "10.xxx.xxx.xxx", "host": "xxx.com", "scheme": "https", "request_method": "GET", "request_uri": "/static/img/logo_new.png", "request_id": "xxxxxxxxxxxxxx", "status": 304, "upstream_addr": "xxx.xx.xx.xx:80", "upstream_status": 304, "request_time": 0.002, "upstream_response_time": 0.000, "upstream_connect_time": 0.000, "upstream_header_time": 0.000}

My Logstash ConfigFile -

input {
        beats {
            port => 50XX
        }
    }
    filter {
        if [kubernetes][container][name] == "nginx" {
            grok {
                match => {
                    "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
                }
                remove_field => "message"
            }
            geoip {
                source => "remote_ip"
                target => "[geoip]"
            }
        }

        date {
            match => ["time", "ISO8601"]
            remove_field => ["time"]
        }
        mutate {
            remove_field => ["source", "host", "[beat][name]", "[beat][version]"]
        }
    }

    output {
            elasticsearch {
                hosts => ["es-xx-01.xxxx.pro:9200", "es-xx-02.xxxx.pro:9200"]
                index => "apps-qa-%{[kubernetes][namespace]}-deployment-%{[kubernetes][pod][name]}-%{[kubernetes][labels][app]}-%{[kubernetes][container][name]}-%{+YYYY.MM.dd}"

        }
    }

How can I config the Logstash the right way, to geet the Message log parsed?

Thanks,

Aleksei

[Badger]

Use a json filter

json { source => "message" }

[aleksei.saiko]

Does it has to be inside the grok, or no inside the grok? Both ways, it didn't work.

[pjanzen (Paul Janzen)]

This is an basic example of a filter that should produce output. Although it is depending on the if statement, if that matches then the below config should work..

input {
        beats {
            port => 50XX
        }
    }
    filter {
        if [kubernetes][container][name] == "nginx" {
            json {
                source => "message"
           }
       }
    }

    output {
            elasticsearch {
                hosts => ["es-xx-01.xxxx.pro:9200", "es-xx-02.xxxx.pro:9200"]
                index => "apps-qa-%{[kubernetes][namespace]}-deployment-%{[kubernetes][pod][name]}-%{[kubernetes][labels][app]}-%{[kubernetes][container][name]}-%{+YYYY.MM.dd}"

        }
    }

[aleksei.saiko]

Thanks, worked