How to Create Fields from Data in JSON Object

Elastic Stack Logstash
1

I have JSON that looks like this:

"httpMessage":{

"method":"GET",
"query":"v=1.020.107",
"start":"1537967152",
"path":"/themes/main/js/jscript.js",
"protocol":"HTTP/1.1",
"requestHeaders":"Host: www.example.com Referer: https://www.example.com/pid/330254684876 Cookie: .ASPXANONYMOUS=0zu1biuM1AEkAAAAMjI4MDIyMzQtN2JlMS00NGMwLWEwODItZjFkODZmOGJiNmUzJL1MheQAQCYvMIQPQYdBIM7Mj4Y1 Accept-Language: en-US Cache-Control: max-age=21679 Connection: keep-alive Accept: / User-Agent: AdsBot-Google ( http://www.google.com/adsbot.html) Accept-Encoding: gzip,deflate,br ",
"responseHeaders":"Content-Type: application/javascript Last-Modified: Wed, 29 Aug 2018 17:34:53 GMT Accept-Ranges: bytes ETag: "e2e38c98be3fd41:0" Server: Microsoft-IIS/8.5 X-Powered-By: ASP.NET Content-Encoding: gzip Content-Length: 26196 Cache-Control: max-age=0 Expires: Wed, 26 Sep 2018 13:05:52 GMT Date: Wed, 26 Sep 2018 13:05:52 GMT Connection: keep-alive Vary: Accept-Encoding ",
"port":"443",
"requestId":"d0ef6h75",
"bytes":"26196",
"host":"www.example.com",
"tls":"tls1.2",
"status":"200"

Logstash will correctly parse the JSON to httpMessage.requestHeaders as

"Host: www.example.com Referer: https://www.example.com/pid/330254684876 Cookie: .ASPXANONYMOUS=0zu1biuM1AEkAAAAMjI4MDIyMzQtN2JlMS00NGMwLWEwODItZjFkODZmOGJiNmUzJL1MheQAQCYvMIQPQYdBIM7Mj4Y1 Accept-Language: en-US Cache-Control: max-age=21679 Connection: keep-alive Accept: / User-Agent: AdsBot-Google ( http://www.google.com/adsbot.html) Accept-Encoding: gzip,deflate,br "

Question is, how can I pull out the individual pieces like "Host", "Referer", "Cookie", etc from within that JSON object into their own fields?? - example: httpMessage.requestHeaders.Host

2

You can use a mutate filter to rename or copy fields to a new location. Just pay attention to how Logstash field references look; httpMessage.requestHeaders.Host must be referenced as [httpMessage][requestHeaders][Host].

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

3

Thanks for your reply. So I added:

add_field => {
"TEST1" => "%{[httpMessage][requestHeaders][Host]}"
}

to my JSON filter section to try to pull that out into its own field.

This is what gets extracted into my TEST1 field: %{[httpMessage][requestHeaders][Host]}.

What am I doing wrong?

EDIT I think the reason it isn't working is that each value within the object does not have quotes around it. For example, it is Host and not "Host". Given this, what would be my workaround?

4

The [httpMessage][requestHeaders] field isn't an object with discrete fields, it's a string with key/value pairs. You can parse it with a kv filter.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.