File not loaded entirely using stdin plugin


(Kml) #1

I am using stdin plugin to load a file of about 2500 events, parse it and then write it to elasticsearch, so what I have is:
$ cat file | ./bin/logstash -f config/myconfig.conf

but what I have is that only about 600 events only are written to elasticsearch.

Afetr that, when I run the command:

$ ./bin/logstash -f config/myconfig.conf

the remaining part of the file is then handled.

do I have a problem with buffering? Can anyone help or expalin what is the problem?


(Kml) #2

The problem is: I had in my logstash.yml:
queue.type: persisted
the default value is memory, when I commented this, it worked.


(Ry Biesemeyer) #3

As a folow-up, enabling queue.drain is a way to resolve this with the persistent queue enabled.

Here's why:

  • when the stdin input reaches EOF, it shuts itself down
  • when a pipeline has no active inputs, it initiates shutdown
  • the memory queue, and the persistent queue in drain-mode, hold the pipeline open until all wueued events are processed
  • the persistent queue without drain-mode allows the pipeline to shut down immediately, since the event's have safely been spooled to disk.

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.