# Extract multiline logs as single event

**URL:** <https://discuss.elastic.co/t/extract-multiline-logs-as-single-event/366545>\
**Category:** Elasticsearch\
**Tags:** painless, ingest-pipeline\
**Created:** [September 13, 2024, 3:24pm UTC](https://discuss.elastic.co/t/extract-multiline-logs-as-single-event/366545 "2024-09-13T15:24:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rubhamra](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@rubhamra](https://discuss.elastic.co/u/rubhamra)\
**Post date:** [September 13, 2024, 3:24pm UTC](https://discuss.elastic.co/t/extract-multiline-logs-as-single-event/366545/1 "2024-09-13T15:24:40Z")

</div>

Hello I am trying to extract java logs , these logs are multiline but I want extract as single event

Ingest pipeline:

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
  "description": "Pipeline to combine multiline logs and remove timestamps",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP:timestamp} %{GREEDYDATA:log_message}"],
        "pattern_definitions": {
          "TIMESTAMP" : "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
        }
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.combined_message == null) {
            ctx.combined_message = ctx.log_message;
          } else {
            ctx.combined_message += '\\n' + ctx.log_message;
          }
        """
      }
    },
    {
      "remove": {
        "field": ["timestamp", "log_message"]
      }
    }
  ]
},
  "docs": [
    {
      "_source": {
        "message": """
Tue Oct 25 15:37:47.449 2022 java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.ActiveMQConnection$5@6ed2142b rejected from java.util.concurrent.ThreadPoolExecutor@6015e781[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
Tue Oct 25 15:37:47.450 2022 at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
Tue Oct 25 15:37:47.459 2022 at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
Tue Oct 25 15:37:47.459 2022 at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
Tue Oct 25 15:37:47.459 2022 at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1964)
Tue Oct 25 15:37:47.459 2022 at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
Tue Oct 25 15:37:47.459 2022 at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
Tue Oct 25 15:37:47.459 2022 at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
Tue Oct 25 15:37:47.459 2022 at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
Tue Oct 25 15:37:47.460 2022 at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
Tue Oct 25 15:37:47.460 2022 at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
Tue Oct 25 15:37:47.460 2022 at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
Tue Oct 25 15:37:47.460 2022 at org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:92)
Tue Oct 25 15:37:47.460 2022 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
Tue Oct 25 15:37:47.460 2022 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Tue Oct 25 15:37:47.460 2022 at java.lang.Thread.run(Unknown Source)"""
      }
    }
  ]
}

```

Expected OUTPUT

```auto
java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.ActiveMQConnection$5@6ed2142b rejected from java.util.concurrent.ThreadPoolExecutor@6015e781[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
	at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1964)
	at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
	at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
	at org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:92)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

```

Thanks

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [September 14, 2024, 8:04pm UTC](https://discuss.elastic.co/t/extract-multiline-logs-as-single-event/366545/2 "2024-09-14T20:04:03Z")

</div>

Hi @rubhamra

Maybe [GSUB](https://www.elastic.co/guide/en/elasticsearch/reference/current/gsub-processor.html) resolve this:

```auto
{
      "gsub": {
        "field": "message",
       "pattern": "\\b(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)\\s(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\\.\\d{3}\\s\\d{4}",
        "replacement": ""
      }
    }

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 15, 2024, 6:48am UTC](https://discuss.elastic.co/t/extract-multiline-logs-as-single-event/366545/3 "2024-09-15T06:48:37Z")

</div>

I don't know how do you send it, by FB or something else, you should treat as multi-line at source side. That is common approach. ES will receive lines as a single event/message.

---

<div class="post-metadata">

**Author:** ![rubhamra](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@rubhamra](https://discuss.elastic.co/u/rubhamra)\
**Post date:** [September 19, 2024, 12:41pm UTC](https://discuss.elastic.co/t/extract-multiline-logs-as-single-event/366545/4 "2024-09-19T12:41:44Z")

</div>

> [@RabBit\_BR](#):
>
> `\\b(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)\\s(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\\.\\d{3}\\s\\d{4}"`

Thank you Rios, that resolved the issue.