# Multiline Plugin - metadata missing from last line

**URL:** <https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725>\
**Category:** Logstash\
**Created:** [June 20, 2018, 3:13pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725 "2018-06-20T15:13:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![MonicaL](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@MonicaL](https://discuss.elastic.co/u/MonicaL)\
**Post date:** [June 20, 2018, 3:13pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725/1 "2018-06-20T15:13:15Z")

</div>

Hi,

I am trying to combine multiple lines into one event using the multiline codec. I also need some filename metadata. The lines are combining properly. However, the metadata of the filename is lost for the last event read from a file.

Here is my configuration:

```auto
input {
        s3{
                bucket => "bucket_name"
		region => "us-east-2"
		codec => multiline {
					pattern => "^(%{DATESTAMP})"
					negate => "true"
					what => "previous"
		}
        }
}
filter {
	mutate { add_field => { "file_name" => "%{[@metadata][s3][key]}"}}
}
output{
		stdout { codec => rubydebug }
}

```

The sample input (sampleLog.txt):

```auto
06-19-2018 15:25:35.7046|ERROR
	more info...
06-19-2018 15:25:35.7046|DEBUG
	more info...
06-19-2018 15:25:35.7046|INFO
	more info...

```

And the logstash output:

```auto
{
    "@timestamp" => 2018-06-20T14:41:09.998Z,
       "message" => "06-19-2018 15:25:35.7046|ERROR\r\n\tmore info...\r",
          "tags" => [
        [0] "multiline"
    ],
      "@version" => "1",
     "file_name" => "sampleLog.txt"
}
{
    "@timestamp" => 2018-06-20T14:41:09.998Z,
       "message" => "06-19-2018 15:25:35.7046|DEBUG\r\n\tmore info...\r",
          "tags" => [
        [0] "multiline"
    ],
      "@version" => "1",
     "file_name" => "sampleLog.txt"
}
{
    "@timestamp" => 2018-06-20T14:41:09.999Z,
       "message" => "06-19-2018 15:25:35.7046|INFO\r\n\tmore info...\r",
          "tags" => [
        [0] "multiline"
    ],
      "@version" => "1",
     "file_name" => "%{[@metadata][s3][key]}"
}

```

I have noted that the filename metadata is missing whether or not the final line was part of a multiline event. Also, I see that if I remove the multiline codec from my configuration, the filename metadata appears for the final line (but then obviously multiline events are not combined).

Any help is much appreciated!

---

<div class="post-metadata">

**Author:** ![MonicaL](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@MonicaL](https://discuss.elastic.co/u/MonicaL)\
**Post date:** [June 20, 2018, 4:46pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725/2 "2018-06-20T16:46:01Z")

</div>

Update: It appears that my problem might have something to do with the use of "what =\> "previous"". If I change "previous" to "next", the last line will include metadata. However, this is not the proper grouping I need for multiline events.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 20, 2018, 5:09pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725/3 "2018-06-20T17:09:01Z")

</div>

It looks like a bug to me. Looking at the [source](https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb), in the main loop of reading the file it sets [metadata][s3][key] on each event before it pushes it onto the queue (line 212). But after the main loop completes, if it flushes the codec then it does not add that key before queueing the event (line 220).

---

<div class="post-metadata">

**Author:** ![MonicaL](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@MonicaL](https://discuss.elastic.co/u/MonicaL)\
**Post date:** [June 22, 2018, 2:54pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725/4 "2018-06-22T14:54:05Z")

</div>

Thanks! I have opened a GitHub issue for this:

> <https://github.com/logstash-plugins/logstash-input-s3/issues/153>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 20, 2018, 2:54pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725/5 "2018-07-20T14:54:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.