Logstash parsing failure

Elastic Stack Logstash
1

I am having an issue getting certain records within my log file to parse and I do not know how to resolve this. Not every record is 1-1 and for records not containing these fields, it works fine and parses\maps properly. Can someone please advise on what can be done to get this working? My config and sample record is below. I am unable to manipulate the log so it is making it hard for me to parse that eventInfo field given the way it is coming in so any help would greatly be appreciated.

// input {
file {
path => "/var/log/logstash/casb_storage/event.log"

start_position => "beginning"

sincedb_path => "/dev/null"

}
}

filter {
grok {
match => { "message" => "<14>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{GREEDYDATA:casb}" }
}
kv {
source => "casb"
field_split => ","
include_brackets => true
recursive => "true"
remove_char_value => """
value_split => "="
}

date {
locale => "en"
match => [ "timestamp", "MMM dd yyyy HH:mm:ss" ]
}

mutate {
  remove_field => [ "casb","message","syslog_timestamp","syslog_hostname","type","path","@index","@version","host","port","tags" ]
}

}

output {
elasticsearch {
hosts => [ "http://xx.xxx.xxx.xxx:xxxx" ]
user => "elastic"
password => "elastic_test_p@ssw0rd"
document_type => "doc"
index => "casb_audit"
} file {
path => "/var/log/logstash/casb_storage/test.log"
}

// <14>Apr 16 13:34:13 server.com auditEventTypeEventCategoryId=180,auditEventTypeEventCategoryName=Incidents,auditEventTypeEventTypeId=2002,auditEventTypeEventTypeName="Change Incident Owner",auditEventTypeSubTypeId=0,createdTime="Apr 16 2019 18:17:16.000 UTC",eventInfo="Changed Owner to John Doe. Incident IDs: 13870, 21264, 21272, 27698, 27700, 27701, 32582, 33131, 37662, 37837, 41246, 43810, 50220, 53428, 59517, 71785, 73560, 73563, 84358, 84360, 84450, 87290, 88713, 88720, 88722, 88723, 88724, 91440, 91530, 91531, 93609, 93760, 99044, 99116, 99267, 99294, 100969, 101211, 111510, 111515, 111509, 111512, 111514, 111526, 111507, 111497, 111500, 111508, 111499, 111523, 111501, 111503, 111504, 111525, 111498, 111506, 111502, 111505, 111527, 111513, 111511, 111520, 111519, 111518, 111522, 111521, 111516, 111517, 111524, 111528, 111530, 111529, 111531, 111534, 111537, 111535, 111532, 111536, 111533, 111540, 111542, 111541, 111539, 111546, 111549, 111548, 111544, 111550, 111547, 111553, 111552, 111555, 111551, 111556, 111558, 111557, 111559, 111561, 111560, 111562, 111564, 111565, 111566, 111563, 111567, 111568, 111569, 111575, 111570, 111573, 111572, 113228, 113229, 113230, 113231, 113232, 113233, 113234, 113235, 113237, 113236, 113238, 113239, 129341, 129426, 135672, 135675, 139416, 139581, 139686, 143504, 145509, 152415, 160709",insertionId=18403852,objectName=BULK,tenantId=72854,timestamp="Apr 16 2019 18:17:13.000 UTC",userInfoEmail=John.Doe@server.com,userInfoFirstName=John,userInfoLastName=Doe,userInfoUserId=36145

2

Your filter works as is for me.

"auditEventTypeEventCategoryName" => "Incidents",
                       "tenantId" => "72854",
        "auditEventTypeSubTypeId" => "0",
      "auditEventTypeEventTypeId" => "2002",
                       "@version" => "1",
    "auditEventTypeEventTypeName" => "Change Incident Owner",
                      "eventInfo" => "Changed Owner to John Doe. Incident IDs: 13870, 21264, 21272, 27698, 27700, 27701, 32582, 33131, 37662, 37837, 41246, 43810, 50220, 53428, 59517, 71785, 73560, 73563, 84358, 84360, 84450, 87290, 88713, 88720, 88722, 88723, 88724, 91440, 91530, 91531, 93609, 93760, 99044, 99116, 99267, 99294, 100969, 101211, 111510, 111515, 111509, 111512, 111514, 111526, 111507, 111497, 111500, 111508, 111499, 111523, 111501, 111503, 111504, 111525, 111498, 111506, 111502, 111505, 111527, 111513, 111511, 111520, 111519, 111518, 111522, 111521, 111516, 111517, 111524, 111528, 111530, 111529, 111531, 111534, 111537, 111535, 111532, 111536, 111533, 111540, 111542, 111541, 111539, 111546, 111549, 111548, 111544, 111550, 111547, 111553, 111552, 111555, 111551, 111556, 111558, 111557, 111559, 111561, 111560, 111562, 111564, 111565, 111566, 111563, 111567, 111568, 111569, 111575, 111570, 111573, 111572, 113228, 113229, 113230, 113231, 113232, 113233, 113234, 113235, 113237, 113236, 113238, 113239, 129341, 129426, 135672, 135675, 139416, 139581, 139686, 143504, 145509, 152415, 160709",
                    "createdTime" => "Apr 16 2019 18:17:16.000 UTC",
               "userInfoLastName" => "Doe",
                    "insertionId" => "18403852",
                      "timestamp" => "Apr 16 2019 18:17:13.000 UTC",
              "userInfoFirstName" => "John"

That said, if these is a field causing you issues you can grok it out and the mutate+gsub it away...

    grok { match => { "message" => 'eventInfo="(?<eventInfo>[^"]*)"' } }
    mutate { gsub => [ "message", 'eventInfo="[^"]*",', "" ] }
3

Thanks Badger-I am surprised it worked for you and not for me. The records that are formatted this way don't even make it over to Kibana and only show @timestamp, _id, _index, and _score. With that being said, I will go ahead and give what you provided a try and see if that does the trick. Thanks again!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.