How to Create Fields from Data in JSON Object

[capsnumber1 (AJ Reeves)]

I have JSON that looks like this:

"httpMessage":{

"method":"GET",
"query":"v=1.020.107",
"start":"1537967152",
"path":"/themes/main/js/jscript.js",
"protocol":"HTTP/1.1",
"requestHeaders":"Host: www.example.com Referer: https://www.example.com/pid/330254684876 Cookie: .ASPXANONYMOUS=0zu1biuM1AEkAAAAMjI4MDIyMzQtN2JlMS00NGMwLWEwODItZjFkODZmOGJiNmUzJL1MheQAQCYvMIQPQYdBIM7Mj4Y1 Accept-Language: en-US Cache-Control: max-age=21679 Connection: keep-alive Accept: / User-Agent: AdsBot-Google ( http://www.google.com/adsbot.html) Accept-Encoding: gzip,deflate,br ",
"responseHeaders":"Content-Type: application/javascript Last-Modified: Wed, 29 Aug 2018 17:34:53 GMT Accept-Ranges: bytes ETag: "e2e38c98be3fd41:0" Server: Microsoft-IIS/8.5 X-Powered-By: ASP.NET Content-Encoding: gzip Content-Length: 26196 Cache-Control: max-age=0 Expires: Wed, 26 Sep 2018 13:05:52 GMT Date: Wed, 26 Sep 2018 13:05:52 GMT Connection: keep-alive Vary: Accept-Encoding ",
"port":"443",
"requestId":"d0ef6h75",
"bytes":"26196",
"host":"www.example.com",
"tls":"tls1.2",
"status":"200"

Logstash will correctly parse the JSON to httpMessage.requestHeaders as

"Host: www.example.com Referer: https://www.example.com/pid/330254684876 Cookie: .ASPXANONYMOUS=0zu1biuM1AEkAAAAMjI4MDIyMzQtN2JlMS00NGMwLWEwODItZjFkODZmOGJiNmUzJL1MheQAQCYvMIQPQYdBIM7Mj4Y1 Accept-Language: en-US Cache-Control: max-age=21679 Connection: keep-alive Accept: / User-Agent: AdsBot-Google ( http://www.google.com/adsbot.html) Accept-Encoding: gzip,deflate,br "

Question is, how can I pull out the individual pieces like "Host", "Referer", "Cookie", etc from within that JSON object into their own fields?? - example: httpMessage.requestHeaders.Host

[magnusbaeck (Magnus Bäck)]

You can use a mutate filter to rename or copy fields to a new location. Just pay attention to how Logstash field references look; httpMessage.requestHeaders.Host must be referenced as [httpMessage][requestHeaders][Host].

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

[capsnumber1 (AJ Reeves)]

Thanks for your reply. So I added:

add_field => {
"TEST1" => "%{[httpMessage][requestHeaders][Host]}"
}

to my JSON filter section to try to pull that out into its own field.

This is what gets extracted into my TEST1 field: %{[httpMessage][requestHeaders][Host]}.

What am I doing wrong?

EDIT I think the reason it isn't working is that each value within the object does not have quotes around it. For example, it is Host and not "Host". Given this, what would be my workaround?

[magnusbaeck (Magnus Bäck)]

The [httpMessage][requestHeaders] field isn't an object with discrete fields, it's a string with key/value pairs. You can parse it with a kv filter.