Logstash is only partially reading input files

Elastic Stack Logstash
1

I'm trying to parse some very large input files (>10,000 lines) with Logstash, pass them through the CSV, ruby, and mutate filters, and send them out via HTTP. My config is something like this:

input {
  file {
    path => "/system1/logs/*"
    type => "sys1"
    sincedb_path => "/system1/sincedb"
    start_position => "beginning"
  }
  file {
    path => "/system2/logs/*"
    type => "sys2"
    sincedb_path => "/system2/sincedb"
    start_position => "beginning"
  }
}
output { http { /*output stuff*/ } }
filter { /* filters */ }

Logstash starts parsing the first files for both systems and pops out a few thousand events via HTTP. Then it stops. The sincedbs are not created, and Logstash doesn't move on (I've waited >30m for it to move on, and it doesn't seem to pop out a single additional event; the first few thousand events take only a minute or two to handle). The log (at "trace" level) ends with a bunch of "output received" lines followed by "Pushing flush onto pipeline"; there seems to be no error message.

What's going on?

2

OK, a sample last entry:

[timestamp][DEBUG][logstash.pipeline  ] output received {"event"=>{...event details...}}
[time][DEBUG][logstash.inputs.file ] Reveived line {:path=>"/system1/logs/log1", :text=>"...text..."}
[time][DEBUG][logstash.inputs.file ] Reveived line {:path=>"/system2/logs/log1", :text=>"...text..."}
[time][DEBUG][logstash.pipeline  ] Pushing flush onto pipeline

This is with the pipeline size set to 1. If I try restarting it, I get warnings that the Mutate filter is hanging.

3

can you post a thread dump or the return of the hot threads api when logstash is stalling?

4

I'll see if I can get the thread dump. In the meantime, I did a couple changes to test things. It now reads the files using a Filebeats instance on the same server, and dropped my filters except a match filter (to ignore comments in the log file), a CSV filter, and a date filter. Now, Logstash no longer says pushing flush onto pipeline; it just ends with an output received line. Filebeat is reporting I/O timeouts. The top threads are Runner, worker0, and worker1, with the latter two's call stacks ending at http.rb:141.

5

Update: Applying this patch fixed the issue.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.