# Multiline Logstash that handles timestamp on each line

**URL:** <https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556>\
**Category:** Logstash\
**Created:** [January 5, 2023, 2:15pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556 "2023-01-05T14:15:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Scotsie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scotsie/32/115614_2.png) [@Scotsie](https://discuss.elastic.co/u/Scotsie)\
**Post date:** [January 5, 2023, 2:15pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/1 "2023-01-05T14:15:53Z")

</div>

I'm currently ingesting logs from multiple devices successfully, one document per row. One particular brand, Polycom, is sending a multiline entry that includes the timestamp for each row.  
**Sample Logging** (with normal and multiline entries)

```auto
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|dns|1|00|doDNSLookupForList(A): returning passed in ipAddress '10.0.0.1'
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sip|0|00|Trying to send data to Destination '10.0.0.1' attempting..
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00|>>>>>>>> REG[1] Data Sent to UDP 10.0.0.1 on socket 189\n
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| REGISTER sip.fqdn\n
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| Via: SIP/2.0/UDP\n 10.20.0.1
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| From: "RRichards"
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| To: <sip:503>
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00|>>>>>>>> REG End of data sent
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|dns|1|00|doDNSLookupForList for record A
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|dns|1|00|doDNSLookupForList(A): returning passed in ipAddress '10.0.0.1'
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sip|0|00|Trying to send data to Destination '10.0.0.1' attempting..
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00|<<<<<<<< REG[1] Data Sent to UDP 10.0.0.1 on socket 189\n
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| REGISTER sip.fqdn\n
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| Via: SIP/2.0/UDP\n 10.20.0.1
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| From: "RRichards"
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00| To: <sip:503>
<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00|<<<<<<<< REG End of data sent

```

I was asked if it's possible to make the log entries for those particular lines combine into a single document. I believe this is possible but most examples refer to logging where newlines begin with spaces or do not begin with timestamps.

I have done some research and reading with examples and attempted to create a multiline codec entry but not quite sure I'm understanding the logic completely or may be missing an obvious point.

**10-external-syslog-input.conf**

```auto
input {
    udp {
        port => 10514
        type => "external_syslogs"
        codec => multiline {
            pattern => "^<%{NONNEGINT}>.*([<>]{8} REG End of data sent|\s{4})"
            what => "previous"
        }
    }
}

```

I've found references where work is done in the filtering side but those seems to be older versions. I haven't tried anything with that as of yet but will provide the filtering.

**10-external-syslog-filter.conf**

```auto
filter {
    if [type] == "external_syslogs" {
        grok {
          match => {
            "message" => [
              # Polycom device
              "^<%{NONNEGINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:host} %{DATA:mac}\|%{DATA:device_timestamp}\|%{DATA:ID}\|%{NONNEGINT:event_class}\|%{NONNEGINT:missed_events}\|%{DATA:event}:*%{GREEDYDATA:syslog_message}",
              "^<%{NONNEGINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} \[%{MAC:mac}\] %{GREEDYDATA:syslog_message}",
              "^<%{NONNEGINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{YEAR} %{GREEDYDATA:syslog_message}"
              ]
            }
          remove_field => "message"
          remove_field => "syslog_pri"
          remove_field => ["[event][original]"]
        }
        date {
          match => ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
    }
}

```

Looking for guidance and/or advice. Links to documentation or articles are greatly appreciated as well.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 5, 2023, 6:31pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/2 "2023-01-05T18:31:18Z")

</div>

> [@Scotsie](#):
>
> I was asked if it's possible to make the log entries for those particular lines combine into a single document.

Should that be three events (the first 8 lines as one, then a single line, then the last 8 lines as one)?

---

<div class="post-metadata">

**Author:** ![Scotsie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scotsie/32/115614_2.png) [@Scotsie](https://discuss.elastic.co/u/Scotsie)\
**Post date:** [January 5, 2023, 6:41pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/3 "2023-01-05T18:41:11Z")

</div>

Hey, thanks for the reply.

> [@Badger](#):
>
> Should that be three events (the first 8 lines as one, then a single line, then the last 8 lines as one)?

Sorry, more context would have been smart.  
The \< or \> repeated 8 times are the start of the entry with the same symbols followed by "REG End of data sent" being the end demarcation.  
In the sample logging I provided I would expect:

- 2 single line entries,
- 1 multiline entry of 6 rows
- 3 single line entries,
- 1 multiline entry of 6 rows

I haven't dug into it much to confirm if the # of lines varies but figured the \<\> symbols would be unique enough.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 5, 2023, 7:38pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/4 "2023-01-05T19:38:15Z")

</div>

You cannot do that using a multiline codec because "does the line contain \<\<\<\<" does not provide enough information to tell you whether you are inside or outside of one of the multiline entries. You could do it using ruby. You must set pipeline.workers 1 and pipeline.ordered true

```
    ruby {
        code => '
            @accumulating ||= false
            @data ||= ""

            line = event.get("[@metadata][restOfLine]")

            if @accumulating
                @data += line + " "
                if line =~ /^[<>]{8}/
                    @accumulating = false
                    event.set("data", @data)
                else
                    event.cancel
                end
            else
                if line =~ /^[<>]{8}/
                    @accumulating = true
                    @data = line
                    event.cancel
                else
                    event.set("data", line)
                end
            end
        '
    }
    date { match => ["[@metadata][timestamp]", "YYYY-MM-dd'T'HH:mm:ssZZ" ] }

```

That will get you seven events like

```
{
   "message" => "<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sip|0|00|Trying to send data to Destination '10.0.0.1' attempting..",
      "data" => "Trying to send data to Destination '10.0.0.1' attempting..",
....
}
{
   "message" => "<182>2022-12-30T16:09:57-06:00 10.20.0.1 64167f000000|1230160957|sipt|0|00|<<<<<<<< REG End of data sent",
      "data" => "<<<<<<<< REG[1] Data Sent to UDP 10.0.0.1 on socket 189\\n REGISTER sip.fqdn\\n Via: SIP/2.0/UDP\\n 10.20.0.1 From: \"RRichards\" To: <sip:503> <<<<<<<< REG End of data sent ",
....
}

```

---

<div class="post-metadata">

**Author:** ![Scotsie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scotsie/32/115614_2.png) [@Scotsie](https://discuss.elastic.co/u/Scotsie)\
**Post date:** [January 5, 2023, 8:18pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/5 "2023-01-05T20:18:33Z")

</div>

> [@Badger](#):
>
> You could do it using ruby. You must set pipeline.workers 1 and pipeline.ordered true

Thanks for that lead and the code. I'll look into that, test and follow up.

I appreciate your time!

---

<div class="post-metadata">

**Author:** ![Scotsie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scotsie/32/115614_2.png) [@Scotsie](https://discuss.elastic.co/u/Scotsie)\
**Post date:** [January 31, 2023, 9:58pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/6 "2023-01-31T21:58:46Z")

</div>

Just a quick follow up after a delay. The users who requested the consolidated entries decided it was fine for one document per line instead and have moved on.  
I have this noted to test 'when I have time' but will close this out and mark @Badger's comment as the solution. Once again thank you for the feedback.

Sincerely,  
Scotsie

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2023, 9:58pm UTC](https://discuss.elastic.co/t/multiline-logstash-that-handles-timestamp-on-each-line/322556/7 "2023-02-28T21:58:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.