Yet another buffer question

Hello All,

I'm indexing local log-files with Logstash + ES and encountered a "buffer" issue: the last messages get sent to ES only after sending Ctrl-C to the LS-process, e.g.:
all logfiles are parsed: ES index - 96,521 hits
waiting and waiting... noting gets sent to ES
Hit Ctrl-C in the Logstash cmd window

22:04:57.939 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
22:04:57.976 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

another events get sent: ES index - 96,919 hits

I've tried to experiment with the following parameters for ES output plugin:

flush_size => 1000
idle_flush_time => 1

but nothing changed: the last batch sits in LS until the process gets a stop command.

LS version: 5.2.2, related setting:

pipeline.workers: 2
pipeline.output.workers: 1
pipeline.batch.size: 200
pipeline.batch.delay: 1000

How could I make LS sending "not full" batches without stopping the process?

What does your config look like?

input {
	file {
		path => "${LOG_DIR}/**/*.log"
		start_position => "beginning"
		sincedb_path => "NUL"
		ignore_older => 0

		codec => multiline {
		  pattern => "^%{TIMESTAMP_ISO8601}\s+"
		  negate => true
		  what => previous

filter {
	# some filters: mutate, grok, date, ruby, drop, anonymize...


output {
	elasticsearch {
		action => "index"
		hosts => "localhost:9200"
		user => "elastic"
		password => "changeme"
		index => "logs-%{+YYYY.MM.dd}"
		document_id => "%{checksum}"
		document_type => "logs"
		template_name => "logs"
		template => "C:/path/to/es-template-logs.json"
		template_overwrite => "true"
		flush_size => 1000
		idle_flush_time => 1
	stdout {
		codec => dots

It's most likely because the multiline codec is waiting for the last line.

You should be able to set the auto_flush_interval in the codec to make this release the last event after a specific period of time instead of waiting indefinitely for a next line.

wow, that codec buffers too! :slight_smile:
Thank you, guys! yes, the multi-line codes was holding data. I've set the parameter auto_flush_interval => 1 (I guess, one second is an appropriate value for a local FS input) and got all records indexed immediately.
Great support, really appreciate that!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.