Multiline codec with file input

Elastic Stack Logstash
1

No, it's not the usual problem. Using 6.2.4 I have a file that contains

1
1

2
2

3

They really are blank lines.

# od -a bar.txt
0000000   1  nl   1  nl  nl   2  nl   2  nl  nl   3  nl

With a stdin input, a multiline codec chops it up just fine. Running

/usr/share/logstash/bin/logstash -e '
    input {
        stdin {
            codec => multiline { pattern => "^$" negate => true what => "previous" }
        }
    }
    output { stdout { codec => rubydebug } }' < bar.txt

Gets me

{
          "host" => "[...]",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
    "@timestamp" => 2018-05-22T20:54:01.262Z,
       "message" => "\n2\n2"
}
{
          "host" => "[...]",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
    "@timestamp" => 2018-05-22T20:54:01.260Z,
       "message" => "1\n1"
}

However using a file input

/usr/share/logstash/bin/logstash -e '
    input {
        file {
            path => "/path/to/bar.txt"
            codec => multiline { pattern => "^$" negate => true what => "previous" }
            start_position => "beginning"
            sincedb_path => "/dev/null"
        }
    }
    output { stdout { codec => rubydebug } }'

Gets me nothing until I SIGINT logstash, at which point it flushes the whole file, with the blank lines deleted.

{
      "@version" => "1",
          "path" => "/path/to/bar.txt",
    "@timestamp" => 2018-05-22T20:56:23.094Z,
          "tags" => [
        [0] "multiline"
    ],
       "message" => "1\n1\n2\n2\n3",
          "host" => "[...]"
}

auto_flush_interval does not fix anything, just saves me having to do the SIGINT. What am I missing?

2

Anyone?

3

Does the file have line terminators?

4

Yes, this is on Linux, so nl is the line terminator.

5

If this is test environment try deleting data/registry and run again

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.