Which algorithms are used in grok filter to find exact match?

Elastic Stack Logstash
1

Hi all, I am using grok filter to find an exact match in logs. My logs size would be 10 mb. And streaming of logs is continuing going on. I am not getting that every time match is searched in 10 mb of log size Will it be good or not.

Algorithms used are efficient or not.

2

It would help if you share your config and provide a more detailed description of what you are looking to do. Logstash by default processes the data in the log file line by line, so will not run grok against the full file.

3

@Christian_Dahlqvist here is the snapshot of logs

rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] [] (kthread+0xe0/0xe4) from [] (ret_from_fork+0x14/0x20)
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
[15119.753902] Dhd_check_hang: Event HANG send up due to re=5 te=0 e=-110 s=2
[15119.753917] Dhd_check_hang: Event HANG send up due to re=5 te=0 e=-110 s=2
[15119.753937] Dhd_prot_ioctl : bus is down. we have nothing to do
[15119.791431] [] (schedule_timeout+0x158/0x25c) from [] (0xea1e0000)
[15119.799331] kworker/3:2 R running 0 29597 2 0x00000000
[15119.805699] [] (__schedule+0x3d0/0x8a4) from [] (worker_thread+0x1fc/0x3dc)
[15119.814384] [] (worker_thread+0x1fc/0x3dc) from [] (kthread+0xe0/0xe4)
[15119.822637] [] (kthread+0xe0/0xe4) from [] (ret_from_fork+0x14/0x20)
[15119.830710] kworker/u8:1 S c0ab2fd4 0 29738 2 0x00000000
[15119.837078] [] (__schedule+0x3d0/0x8a4) from [] (worker_thread+0x1fc/0x3dc)
[15119.845763] [] (worker_thread+0x1fc/0x3dc) from [] (kthread+0xe0/0xe4)
[15119.854015] [] (kthread+0xe0/0xe4) from [] (ret_from_fork+0x14/0x20)
[15119.862088] kworker/u8:4 S c0ab2fd4 0 29739 2 0x00000000
[15119.868455] [] (__schedule+0x3d0/0x8a4) from [] (worker_thread+0x1fc/0x3dc)
[15119.877140] [] (worker_thread+0x1fc/0x3dc) from [] (kthread+0xe0/0xe4)
[15119.885391] [] (kthread+0xe0/0xe4) from [] (ret_from_fork+0x14/0x20)
[15119.893468] Sched Debug Version: v0.10, 3.10.96+ #1
[15119.898337] ktime

And filter used is

grok
{
match => { logs => "dhd_check_hang: Event HANG send up due to"}
}

4

What is the expected output/result?

5

expected output result is

[15119.753902] Dhd_check_hang: Event HANG send up due to re=5 te=0 e=-110 s=2
[15119.753917] Dhd_check_hang: Event HANG send up due to re=5 te=0 e=-110 s=2

6

Are you looking to filter out all lines that does not match that pattern? What do you want the resulting document being sent to Elasticsearch to look like?

7

I am using grok filter to find regex match in logs and extract only those line and dumps these lines in ES

8

Would something like this work?

if [message] !~ /Dhd_check_hang: Event HANG send up due to/ {
  drop {}
}
9

If it will check line by line then it will take more time to finad match and dump

10

If it will check line by line then it will take more time to finad match and dump

Maybe. Have you measured?

The background of your questions is unclear. What are you getting at?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.