Anyone else seeing this behavior:
Logstash 6.1.0 and 6.1.1 is not reading entire stdin. Logstash 6.0.x works correctly.
Test conditions:
Create an input file '5000recs.json' containing newline delimited JSON records like these:
{"rec": 1, "field1": "this is some text", "field2": "this is some more text"}
{"rec": 2, "field1": "this is some text", "field2": "this is some more text"}
{"rec": 3, "field1": "this is some text", "field2": "this is some more text"}
{"rec": 4, "field1": "this is some text", "field2": "this is some more text"}
...
{"rec": 4999, "field1": "this is some text", "field2": "this is some more text"}
{"rec": 5000, "field1": "this is some text", "field2": "this is some more text"}
Run logstash on command line like this:
$ logstash -e 'input {stdin {codec=>json_lines}} output {stdout {codec=>rubydebug}}' < 5000recs.json
Monitor stdout and see how many records get processed before the pipeline terminates.
With logstash 6.1.0 and 6.1.1 the pipeline terminates after processing less than ~ 4,500 records and the exact point varies from run to run. Example: on four consecutive runs it stopped at 4456 records on the first run, 4250 on the second, 4319 on the third, and 4250 on the fourth. There are no error messages on either the command line or in the logstash log files.
With logstash 6.0.0 and 6.0.1 it consistently processes all 5000 records and then terminates.
If you use an input file with 10000 records, 6.1.x processes up to ~ 9439 records and then terminates (for four runs it stopped at 9439, 9183, 9439, 9183 recs), while 6.0.1 always processes the entire 10000 records.