Yet another buffer question

[jetnet]

Hello All,

I'm indexing local log-files with Logstash + ES and encountered a "buffer" issue: the last messages get sent to ES only after sending Ctrl-C to the LS-process, e.g.:
all logfiles are parsed: ES index - 96,521 hits
waiting and waiting... noting gets sent to ES
Hit Ctrl-C in the Logstash cmd window

22:04:57.939 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
22:04:57.976 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

another events get sent: ES index - 96,919 hits

I've tried to experiment with the following parameters for ES output plugin:

flush_size => 1000
idle_flush_time => 1

but nothing changed: the last batch sits in LS until the process gets a stop command.

LS version: 5.2.2, related setting:

pipeline.workers: 2
pipeline.output.workers: 1
pipeline.batch.size: 200
pipeline.batch.delay: 1000

How could I make LS sending "not full" batches without stopping the process?
Thanks!

[warkolm (Mark Walkom)]

What does your config look like?

[jetnet]

input {
	file {
		path => "${LOG_DIR}/**/*.log"
		start_position => "beginning"
		sincedb_path => "NUL"
		ignore_older => 0

		codec => multiline {
		  pattern => "^%{TIMESTAMP_ISO8601}\s+"
		  negate => true
		  what => previous
		}		
	}
}

filter {
	# some filters: mutate, grok, date, ruby, drop, anonymize...

}

output {
	elasticsearch {
		action => "index"
		hosts => "localhost:9200"
		user => "elastic"
		password => "changeme"
		index => "logs-%{+YYYY.MM.dd}"
		document_id => "%{checksum}"
		document_type => "logs"
		
		template_name => "logs"
		template => "C:/path/to/es-template-logs.json"
		template_overwrite => "true"
		
		flush_size => 1000
		idle_flush_time => 1
	}
	stdout {
		codec => dots
	}	
}

[warkolm (Mark Walkom)]

It's most likely because the multiline codec is waiting for the last line.

[Christian_Dahlqvist]

You should be able to set the auto_flush_interval in the codec to make this release the last event after a specific period of time instead of waiting indefinitely for a next line.

[jetnet]

wow, that codec buffers too!
Thank you, guys! yes, the multi-line codes was holding data. I've set the parameter auto_flush_interval => 1 (I guess, one second is an appropriate value for a local FS input) and got all records indexed immediately.
Great support, really appreciate that!