Timeouts for meraki filters 7.4.2

Elastic Stack Logstash
1

Hi guys i am running into issues with the latest 7.4 logstash. my meraki filters are timing out that the message value is too large. I have tried to set a greedydata on the last field type and others. i am sending these to elasticsearch with the output filter.

different net flows

MERAKIALERT %{NOTSPACE}%{SPACE}%{NOTSPACE:[serial]}%{SPACE}%{NOTSPACE:[logsource]}%{SPACE}flows%{NOTSPACE:[netflows]}%{SPACE}%{NOTSPACE:[otherthing]}%{SPACE}src=%{NOTSPACE:[src_ip]}%{SPACE}dst=%{NOTSPACE:[dst_ip]}%{SPACE}mac=%{NOTSPACE:[mac]}%{SPACE}protocol=%{NOTSPACE:[protocol]}%{SPACE}sport=%{NOTSPACE:[src_port]}%{SPACE}dport=%{NOTSPACE:[dst_port]}

MERAKIALERTFLOWS %{NOTSPACE}%{SPACE}%{NOTSPACE:[serial]}%{SPACE}%{NOTSPACE:[logsource]}%{SPACE}%{NOTSPACE}%{SPACE}src=%{NOTSPACE:[src_ip]}%{SPACE}dst=%{NOTSPACE:[dst_ip]}%{SPACE}mac=%{NOTSPACE:[mac]}%{SPACE}protocol=%{NOTSPACE:[protocol]}%{SPACE}sport=%{NOTSPACE:[src_port]}%{SPACE}dport=%{NOTSPACE:[dst_port]}%{SPACE}%{NOtSPACE:[data]}

MERAKIALERTPATTERN %{NOTSPACE}%{SPACE}%{NOTSPACE:[serial]}%{SPACE}%{NOTSPACE:[logsource]}%{SPACE}%{NOTSPACE}%{SPACE}src=%{NOTSPACE:[src_ip]}%{SPACE}dst=%{NOTSPACE:[dst_ip]}%{SPACE}protocol=%{NOTSPACE:[protocol]}%{SPACE}sport=%{NOTSPACE:[src_port]}%{SPACE}dport=%{NOTSPACE:[dst_port]}%{SPACE}%{NOtSPACE:[pattern]}

MERAKIALERTREQUEST %{NOTSPACE}%{SPACE}%{NOTSPACE:[serial]}%{SPACE}%{NOTSPACE:[logsource]}%{SPACE}%{NOTSPACE}%{SPACE}src=%{NOTSPACE:[src]}%{SPACE}dst=%{NOTSPACE:[dst]}%{SPACE}mac=%{NOTSPACE}%{SPACE}%:{URIPATHPARAM:[request]}

MERAKITRANSLATED %{NOTSPACE}%{SPACE}%{NOTSPACE:[serial]}%{SPACE}%{NOTSPACE:[logsource]}%{SPACE}src=%{NOTSPACE:[src_ip]}%{SPACE}dst=%{NOTSPACE:[dst_ip]}%{SPACE}protocol=%{NOTSPACE:[protocol]}%{SPACE}sport=%{NOTSPACE:[src_port]}%{SPACE}dport=%{NOTSPACE:[dst_port]}%{SPACE}translated_dst_ip=%{NOTSPACE:[trans_dst_ip]}%{SPACE}translated_port=%{NOTSPACE:[trans_dst_port]}

][main] Timeout executing grok '%{MERAKIALERTFLOWS}' against field 'message' with value 'Value too large to output (288 bytes)! First 255 chars are: <134>1 1574446937.334809909 PHX_MX250_1 urls src=172.24.1.8:52627 dst=143.166.156.113:80 mac=A4:93:4C:C0:C1:C5 agent='urlgrabber/3.9.1 yum/3.2.29' request: GET http://linux.dell.com/repo/hardware/latest/mirrors.cgi?osname=el6&basearch=x86_64&native=1&dells'!

2

Two things. Do you expect the incoming message to look like

<134>1 1574446937.334809909 PHX_MX250_1 urls src=172.24.1.8:52627 dst=143.166.156.113:80 mac=A4:93:4C:C0:C1:C5 ...

That is, are you matching the entire line? If so, read this and anchor your patterns using ^, like this:

MERAKIALERTFLOWS ^%{NOTSPACE}%{SPACE}%{NOTSPACE:[serial]}%{SPACE}%{NOTSPACE:[logsource]}%{SPACE}%{NOTSPACE}%{SPACE}src=%{NOTSPACE:[src_ip]}%{SPACE}dst=%{NOTSPACE:[dst_ip]}%{SPACE}mac=%{NOTSPACE:[mac]}%{SPACE}protocol=%{NOTSPACE:[protocol]}%{SPACE}sport=%{NOTSPACE:[src_port]}%{SPACE}dport=%{NOTSPACE:[dst_port]}%{SPACE}%{NOtSPACE:[data]}

They will then fail quickly instead of timing out. It is timing out because it is trying to match a [message] field that looks like

Value too large to output (288 bytes)! First 255 chars are: <134>1 1574446937.334809909 PHX_MX250_1 urls src=172.24.1.8:52627 dst=143.166.156.113:80 ...

So it does all the NOTSPACE/SPACE matching at the first character of the string, fails to match, then tries again at the second character of the string, fails to match, etc., etc.

Your problem is upstream of your grok, whatever is generating the [message] field is having a problem.

3

ok Ill try that

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.