Nested Json to parse

Elastic Stack Logstash
1

Hi all again,

I know this topic has been discussed in tha past, but I am a newbie in this world and could not be able to apply to my case the solutions provided in the old discussions.
I have to parse a log like this:

> Sep 22 15:08:11 host04 PWM {"sourceAddress":"10.10.10.5","sourceHost":"10.10.10.5","type":"USER","eventCode":"TOKEN_ISSUED","guid":"3792e530-c41a-4d49-8ac0-d516c9","timestamp":"2017-09-22T13:08:11Z","message":"{\"date\":\"2017-09-22T13:08:11Z\",\"name\":\"NEWUSER_EMAIL\",\"data\":**{\"_______profileID\":\"WRT\",\"CompanyexternalCountry\":\"cn=AGO,ou=countries,ou=mapping,o=xxx\",\"CompanyexternalBrand\":\"cn=Company,ou=brands,ou=mapping,o=xxx\",\"telephoneNumber\":\"\",\"mail\":\"username@nomail.com\",\"CompanyexternalNote\":\"\",\"givenName\":\"john\",\"CompanyexternalCompany\":\"nosocieta\",\"sn\":\"doe\",\"cn\":\"JOHN DOE\",\"Companyexternalvatnumber\":\"34565432\",\"password1\":\"H4sIAAAAAAAAAAFCAL3_UFdDMjU23suete7Z1gFDva07-7WHvegvewZfvVC2DCEsVew4kAIHzQI9-AebILK9uY0un0E1OQlgAEIAAAA=\"}**,\"dest\":[[truncated]","narrative":"A token has been issued","xdasTaxonomy":"XDAS_AE_SET_CRED_ACCOUNT","xdasOutcome":"XDAS_OUT_PRIV_GRANTED"}

The problem is what is after the word "data": i am not capable of mapping and parsing that message inside the {}...

in my logstash.conf I tried to put:

filter {
  if [type] =="syslog" {
 grok {
      match =>[ "message", "%{CISCOTIMESTAMP:timestamp} %{HOSTNAME:Hostname} %{WORD:Application} %{GREEDYDATA:request1}" ]
        }
 json{
        source => "request1"
        target => "WRTuser"
        remove_field=>["request1"]
    }
 json {
        source => "WRTuser.message"
        target => "data"
  }
}
}

(2 json filters one after another)

I was able to parse all the fields outside the inner message, but nothing inside those {}.

Anyone have an idea to suggest?

thanks a lot

2

You're using the wrong syntax to address nested fields, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

3

I've changed the second Json parse to:

json {
        source => "[WRTuser][message]"
        target => "data"
  }

but looks like nothing is changed

4

As far as I can see, the WRTuser.message field is not valid JSON as the data field have ** surrounding the value.

5

Sorry dunno where those ** came from.

another log (today's)

Sep 25 11:14:44 hostname PWM {"sourceAddress":"10.10.10.1","sourceHost":"10.10.10.1","type":"USER","eventCode":"TOKEN_ISSUED","guid":"7f38b26c-65c5-4621-816a-b492cb6c5","timestamp":"2017-09-25T09:14:44Z","message":"{\"date\":\"2017-09-25T09:14:44Z\",\"name\":\"NEWUSER_EMAIL\",\"data\":{\"_______profileID\":\"WRT\",\"CompanyexternalCountry\":\"cn=AND,ou=countries,ou=mapping,o=com\",\"CompanyexternalBrand\":\"cn=Company,ou=brands,ou=mapping,o=com\",\"telephoneNumber\":\"\",\"mail\":\"bncnv@nomail.com\",\"CompanyexternalNote\":\"\",\"givenName\":\"Bianca\",\"CompanyexternalCompany\":\"snowwhite\",\"sn\":\"Neve\",\"cn\":\"BIANCA NEVE\",\"Companyexternalvatnumber\":\"3443322\",\"password1\":\"H4sIAAAAAAAAAAFCAL3_UFdNLkFFUzEyOF9ITUFDMjU23suete7Z1gFDva07-7WHvegvewZfvVC2DCEsVew4kAIHzQI9-AebILK9uY0un0E1OQlgAEIAAAA=\"},\"dest\":[\"b[truncated]","narrative":"A token has been issued","xdasTaxonomy":"XDAS_AE_SET_CRED_ACCOUNT","xdasOutcome":"XDAS_OUT_PRIV_GRANTED"}

6

It does not look like that is well-formed JSON in the message field either.

7

with an online parser, looks like it's correctly parsed if pasted starting from the {

8

I parsed the initial JSON, and this gave me the following string for the message field, which is not valid JSON:

"{\"date\":\"2017-09-25T09:14:44Z\",\"name\":\"NEWUSER_EMAIL\",\"data\":{\"_______profileID\":\"WRT\",\"CompanyexternalCountry\":\"cn=AND,ou=countries,ou=mapping,o=com\",\"CompanyexternalBrand\":\"cn=Company,ou=brands,ou=mapping,o=com\",\"telephoneNumber\":\"\",\"mail\":\"bncnv@nomail.com\",\"CompanyexternalNote\":\"\",\"givenName\":\"Bianca\",\"CompanyexternalCompany\":\"snowwhite\",\"sn\":\"Neve\",\"cn\":\"BIANCA NEVE\",\"Companyexternalvatnumber\":\"3443322\",\"password1\":\"H4sIAAAAAAAAAAFCAL3_UFdNLkFFUzEyOF9ITUFDMjU23suete7Z1gFDva07-7WHvegvewZfvVC2DCEsVew4kAIHzQI9-AebILK9uY0un0E1OQlgAEIAAAA=\"},\"dest\":[\"b[truncated]"

It starts with a {, but does not end with a }.

9

Ok at the end I addressed the problem. My application had a configuration parameter that truncated the "inner" json message after 900 chars. I changed with 2048 and now the json is correctly formatted.

thanks a lot

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.