How to generate single object from multiple log lines in logstash filter?

Elastic Stack Logstash
1

Hi,
I am new to Logstash and Grok filter. I want to parse logs like these -

2018-01-11 17:17:16,071 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | CommittedVirtualMemorySize :: 401186816 
2018-01-11 17:17:16,071 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | FreePhysicalMemorySize :: 1751130112 
2018-01-11 17:17:16,072 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | FreeSwapSpaceSize :: 4294967295 
2018-01-11 17:17:16,694 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | ProcessCpuLoad :: -1.0 
2018-01-11 17:17:16,694 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | ProcessCpuTime :: 47471104300 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | SystemCpuLoad :: 1.0 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | TotalPhysicalMemorySize :: 4285849600 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | TotalSwapSpaceSize :: 4294967295

to an object like this -

{
  "timestamp": "2018-01-11 17:17:16,071",
  "log_level": "DEBUG",
  "thread_name": "Thread-2",
  "class": "com.example.monitor.MonitorHelper",
  "method": "cpuMonitoring",
  "line_number": "307",
  "CommittedVirtualMemorySize": "401186816",
  "FreePhysicalMemorySize": "1751130112",
  "FreeSwapSpaceSize": "4294967295",
  "ProcessCpuLoad": "-1.0",
  "ProcessCpuTime": "47471104300",
  "SystemCpuLoad": "1.0",
  "TotalPhysicalMemorySize": "4285849600",
  "TotalSwapSpaceSize": "4294967295"
}

As of now my grok pattern is -

%{TIMESTAMP_ISO8601:timestamp} \| %{LOGLEVEL:log_level} \| \[(?<thread_name>\b[\w\-]+\b)\] \| %{JAVAFILE:class}\:%{JAVAMETHOD:method}\(%{NUMBER:line_number}\) \| %{GREEDYDATA:log_message}

which provides multiple output lines for each input log line. JSON object looks like this-

{
  "timestamp": "2018-01-11 17:17:16,071",  
  "log_level": "DEBUG",
  "thread_name": "Thread-2",
  "class": "com.example.monitor.MonitorHelper",
  "method": "cpuMonitoring",
  "line_number": "307",
  "log_message": "CommittedVirtualMemorySize :: 401186816 "
}

can you please help me with what I need to look for in order to achieve this?

2

The aggregate filter should be able to help you.

1 Like
3

Thanks for helping me out and responding. I will look into aggregate filter.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.