Logstash not recognizing the JSON objects from my logs and displays entire log set in message field

Elastic Stack Logstash
1

I need to send below logs using logstash and want to create a dashboard using request_header_x_forwarded_for and request_username fields which are JSON objects

Dec 26 00:32:50 Web1 mars-auth-capture: [INFO ][mars][auth-capture]@cee: {"profile":"http://cee.mitre.org/1.0-beta1/coreprofile","host":"1.2.3.4","pname":"mars-auth-capture","native":{"request_timestamp":"12/26/18 00:20:46","request_protocol":"HTTP/1.1","request_method":"POST","request_header_x_forwarded_for":"a.b.c.d","request_username":"sample_username","response_header_set_cookie":"rdAc6opdpQoEmV4hnQogPM","response_auth_result":"error occurred"}}

I tried using JSON filter and Grok filter but every time my entire log set is coming in the message field and it's not being split.

Can someone help as I am new to ELK and need some guidance.

Can someone help?

2

I would use dissect to chop it up then a json filter.

filter {
        dissect {
        mapping => { "message" => "%{ts} %{+ts} %{+ts} %{a} %{b}: [%{level}][%{c}][%{d}@%{e}: %{msg}" }
        }
}
filter { json { source => "msg" } }
3

Thanks a lot.

4

Thanks a lot for your last response. I was able to successfully parse my logs the way they needed to be. Unfortunately the fields I need are tagged as type:text and I am not able to create any dashboard using them (since they are not visible in fields in Visualize Tab). What I understood from the web is that field should be of type keyword to be able to use them to create dashboards. Can you provide me the quick solution on how I can send my logs so that field i need are of type keyboard.

5

If elasticsearch is ingesting fields as text and you want them to be keyword then I think that is an elasticsearch question, not a logstash question.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.