Logstash Parsing Issue - Community Input request

[andynz2017 (Andrew)]

Hi All,

Got a bit of a curly issue which I'm reaching out to the community to help resolve.

I have parsed out a field within Exchange 2010 Transport log which appears to be presentative of a string, but contains both a substring that I wish to convert to unique date field with unique name as well as split the two non-date values using the ";" delimiter and store in the source-context field as sub-fields (I hope I've got the lingo right here).

Field Name: Source-Context
Field Value: 08D48EEB952EE1A6;2017-05-06T06:28:04.718Z;0

I have attempted a mutate on that field using split ";" but I get a field mismatch in logstash-plain.log file and the message is lost. Is there a simple/graceful way of achieving what I want here? The original message uses grok pattern to parse the original message and I convert a lot of the fields to appropriate type depending on usage.

The other field that I believe has a similar issue is

Field Name: message-info
Field Value: 2017-05-06T06:27:56.113Z;SRV=excsvr2010.test.domain.com:TOTAL=8|SMR=8

Ideally I would want to also split this field and convert the date time field and then split out the other SRV, TOTAL and SMR values into their own fields. As the format isn't consistent, I'm finding it difficult to mutate the data to a state that is storable and ability to be queried.

Is there a mutate solution to this problem or would this potentially require an extra grok filter that applies to the two fields above? Keeping in mind that the date contained in these fields may vary in terms of length and the field delimiters may not be consistent.

Thought I would run this question past the community just in case I'm missing a simple fix here.

Any help or guidance is hugely appreciated.

Cheers,
Andrew

[magnusbaeck (Magnus Bäck)]

For the first field I'd either use a grok filter to split it or use the mutate filter's split option, e.g. like this:

grok {
  match => {
    "Source-Context" => "^%{BASE16NUM:whatever};%{GREEDYDATA:timestamp}"
  }
}

You should be able to use something similar for the message-info field, possibly in conjunction with a kv filter. What's ultimately best depends on how the data is inconsistent.

[andrewh]

Hi Magnus,

Thanks for the quick and insightful response. I’ll play around with the different combinations and see what works best – will need to take a number of samples for each to ensure I get the best match.

Thanks for providing sanity around this for me, awesome service.

Cheers,
Andrew