Inserting several grok filters in one logstash conf file

Elastic Stack Logstash
1

Hello guys,
I'm having an error when adding a grok filter to my logstash config:
[FATAL] 2019-05-03 09:53:36.256 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, {, ,, ] at line 7, column 17 (byte 140) after filter {

I've tried to count every "{" in the file, but it doesn't look like there is any missing. Are you somehow able to help me identify the issue?
This is my file for the check:
filter {
grok {
overwrite => ["log"]
break_on_match => true
remove_field => ["log"]
match => [
"log","{"log":"{\"v\":\"%{GREEDYDATA:v}\",\"category\":\"%{GREEDYDATA:category}\",\"level\":\"%{GREEDYDATA:level}\",\"timestamp\":\"%{GREEDYDATA:timestamp}\",\"application\":{\"name\":\"%{GREEDYDATA:name}\",\"version\":\"%{GREEDYDATA:version}\",\"component\":\"%{GREEDYDATA:component}\",\"thread\":\"%{GREEDYDATA:thread}\",\"instanceId\":\"%{GREEDYDATA:instanceId}\",\"process\":\"%{GREEDYDATA:process}\"},\"context\":{\"sessionId\":\"%{GREEDYDATA:sessionId}\",\"tenantId\":\"%{GREEDYDATA:tenantId}\",\"principalId\":\"%{GREEDYDATA:principalId}\",\"ext\":{\"CorrelationId\":\"%{GREEDYDATA:CorrelationId}\",\"x-ol-channel\":\"%{GREEDYDATA:x-ol-channel}\",\"x-ol-correlation-id\":\"%{GREEDYDATA:x-ol-correlation-id}\",\"jti\":\"%{GREEDYDATA:jti}\",\"scenarioName\":\"%{GREEDYDATA:scenarioName}\",\"scenarioId\":\"%{GREEDYDATA:scenarioId}\",\"connector\":\"%{GREEDYDATA:connector}\",\"taskId\":\"%{GREEDYDATA:taskId}\"}},\"details\":{\"type\":\"%{GREEDYDATA:type}\",\"description\":\"%{GREEDYDATA:description}\",\"details\":{\"billing\":\"%{GREEDYDATA:billing}\",\"status\":\"%{GREEDYDATA:status}\"}}}\n","stream":%{GREEDYDATA:stream},"time":"%{GREEDYDATA:time}"}",
"log","%{GREEDYDATA:message}"
]
}
}

Thanks in advance,
Gerardo

2

match is expecting a hash not an array.

So match is expecting a { instead of a [ after =>

3

Hi,
thanks for the reply. How should i then rewrite the filter?
I mean, just by replacing the brackets with the { doesn't solve the issue...
BR
Gerardo

4

That grok pattern is a bit hard to follow but something like this

grok {
  match => { "log" =>  (("{\"v\":\"%{GREEDYDATA:v}\",\"category\":\"%{GREEDYDATA:category}\",\"level\":\"%{GREEDYDATA:level}\",\"timestamp\":\"%{GREEDYDATA:timestamp}\",\"application\":{\"name\":\"%{GREEDYDATA:name}\",\"version\":\"%{GREEDYDATA:version}\",\"component\":\"%{GREEDYDATA:component}\",\"thread\":\"%{GREEDYDATA:thread}\",\"instanceId\":\"%{GREEDYDATA:instanceId}\",\"process\":\"%{GREEDYDATA:process}\"},\"context\":{\"sessionId\":\"%{GREEDYDATA:sessionId}\",\"tenantId\":\"%{GREEDYDATA:tenantId}\",\"principalId\":\"%{GREEDYDATA:principalId}\",\"ext\":{\"CorrelationId\":\"%{GREEDYDATA:CorrelationId}\",\"x-ol-channel\":\"%{GREEDYDATA:x-ol-channel}\",\"x-ol-correlation-id\":\"%{GREEDYDATA:x-ol-correlation-id}\",\"jti\":\"%{GREEDYDATA:jti}\",\"scenarioName\":\"%{GREEDYDATA:scenarioName}\",\"scenarioId\":\"%{GREEDYDATA:scenarioId}\",\"connector\":\"%{GREEDYDATA:connector}\",\"taskId\":\"%{GREEDYDATA:taskId}\"}},\"details\":{\"type\":\"%{GREEDYDATA:type}\",\"description\":\"%{GREEDYDATA:description}\",\"details\":{\"billing\":\"%{GREEDYDATA:billing}\",\"status\":\"%{GREEDYDATA:status}\"}}}\n","stream":%{GREEDYDATA:stream},"time":"%{GREEDYDATA:time}")|%{GREEDYDATA:message}")}
  remove_field => ["log"]
  break_on_match => true
}

In your config above you are also both overwriting and removing the field log. I chose to just remove it in my example...

I assumed you wanted to try to match the log field to either the "complex" pattern or put everything in message so I put the "complex" pattern inside () and added an or (the pipe) so that if the "complex" pattern is not a match, everything is put in message.

I did not test this and I did not check you complex pattern for syntax errors. There is a lot going on...

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.