Extract multiline logs as single event

Elastic Stack Elasticsearch
,
1

Hello I am trying to extract java logs , these logs are multiline but I want extract as single event

Ingest pipeline:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
  "description": "Pipeline to combine multiline logs and remove timestamps",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP:timestamp} %{GREEDYDATA:log_message}"],
        "pattern_definitions": {
          "TIMESTAMP" : "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
        }
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.combined_message == null) {
            ctx.combined_message = ctx.log_message;
          } else {
            ctx.combined_message += '\\n' + ctx.log_message;
          }
        """
      }
    },
    {
      "remove": {
        "field": ["timestamp", "log_message"]
      }
    }
  ]
},
  "docs": [
    {
      "_source": {
        "message": """
Tue Oct 25 15:37:47.449 2022 java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.ActiveMQConnection$5@6ed2142b rejected from java.util.concurrent.ThreadPoolExecutor@6015e781[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
Tue Oct 25 15:37:47.450 2022 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
Tue Oct 25 15:37:47.459 2022 	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
Tue Oct 25 15:37:47.459 2022 	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
Tue Oct 25 15:37:47.459 2022 	at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1964)
Tue Oct 25 15:37:47.459 2022 	at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
Tue Oct 25 15:37:47.459 2022 	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
Tue Oct 25 15:37:47.459 2022 	at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
Tue Oct 25 15:37:47.459 2022 	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
Tue Oct 25 15:37:47.460 2022 	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
Tue Oct 25 15:37:47.460 2022 	at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
Tue Oct 25 15:37:47.460 2022 	at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
Tue Oct 25 15:37:47.460 2022 	at org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:92)
Tue Oct 25 15:37:47.460 2022 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
Tue Oct 25 15:37:47.460 2022 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Tue Oct 25 15:37:47.460 2022 	at java.lang.Thread.run(Unknown Source)"""
      }
    }
  ]
}

Expected OUTPUT

java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.ActiveMQConnection$5@6ed2142b rejected from java.util.concurrent.ThreadPoolExecutor@6015e781[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
	at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1964)
	at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
	at org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:92)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Thanks

2

Hi @rubhamra

Maybe GSUB resolve this:

{
      "gsub": {
        "field": "message",
       "pattern": "\\b(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)\\s(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\\.\\d{3}\\s\\d{4}",
        "replacement": ""
      }
    }
3

I don't know how do you send it, by FB or something else, you should treat as multi-line at source side. That is common approach. ES will receive lines as a single event/message.

4

Thank you Rios, that resolved the issue.

1 Like