# Yet another buffer question

**URL:** <https://discuss.elastic.co/t/yet-another-buffer-question/78259>\
**Category:** Logstash\
**Created:** [March 11, 2017, 9:24pm UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259 "2017-03-11T21:24:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\
**Post date:** [March 11, 2017, 9:24pm UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/1 "2017-03-11T21:24:05Z")

</div>

Hello All,

I'm indexing local log-files with Logstash + ES and encountered a "buffer" issue: the last messages get sent to ES only after sending `Ctrl-C` to the LS-process, e.g.:  
all logfiles are parsed: ES index - **96,521** hits  
waiting and waiting... noting gets sent to ES  
Hit `Ctrl-C` in the Logstash cmd window

```
22:04:57.939 [SIGINT handler] WARN logstash.runner - SIGINT received. Shutting down the agent.
22:04:57.976 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}

```

another events get sent: ES index - **96,919** hits

I've tried to experiment with the following parameters for ES output plugin:

```
flush_size => 1000
idle_flush_time => 1

```

but nothing changed: the last batch sits in LS until the process gets a stop command.

LS version: 5.2.2, related setting:

```
pipeline.workers: 2
pipeline.output.workers: 1
pipeline.batch.size: 200
pipeline.batch.delay: 1000

```

How could I make LS sending "not full" batches without stopping the process?  
Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 11, 2017, 9:33pm UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/2 "2017-03-11T21:33:09Z")

</div>

What does your config look like?

---

<div class="post-metadata">

**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\
**Post date:** [March 11, 2017, 10:17pm UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/3 "2017-03-11T22:17:34Z")

</div>

```
input {
	file {
		path => "${LOG_DIR}/**/*.log"
		start_position => "beginning"
		sincedb_path => "NUL"
		ignore_older => 0

		codec => multiline {
		  pattern => "^%{TIMESTAMP_ISO8601}\s+"
		  negate => true
		  what => previous
		}		
	}
}

filter {
	# some filters: mutate, grok, date, ruby, drop, anonymize...

}

output {
	elasticsearch {
		action => "index"
		hosts => "localhost:9200"
		user => "elastic"
		password => "changeme"
		index => "logs-%{+YYYY.MM.dd}"
		document_id => "%{checksum}"
		document_type => "logs"
		
		template_name => "logs"
		template => "C:/path/to/es-template-logs.json"
		template_overwrite => "true"
		
		flush_size => 1000
		idle_flush_time => 1
	}
	stdout {
		codec => dots
	}	
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 12, 2017, 12:34am UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/4 "2017-03-12T00:34:59Z")

</div>

It's most likely because the multiline codec is waiting for the last line.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 12, 2017, 1:50am UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/5 "2017-03-12T01:50:22Z")

</div>

You should be able to set the [auto\_flush\_interval](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-auto_flush_interval) in the codec to make this release the last event after a specific period of time instead of waiting indefinitely for a next line.

---

<div class="post-metadata">

**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\
**Post date:** [March 12, 2017, 6:57am UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/6 "2017-03-12T06:57:58Z")

</div>

wow, that codec buffers too! 🙂  
Thank you, guys! yes, the multi-line codes was holding data. I've set the parameter `auto_flush_interval => 1` (I guess, one second is an appropriate value for a local FS input) and got all records indexed immediately.  
Great support, really appreciate that!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2017, 6:58am UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259/7 "2017-04-09T06:58:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.