# \[solved\] \[accuracy\] randomly missing events

**URL:** <https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813>\
**Category:** Logstash\
**Created:** [November 17, 2015, 2:29pm UTC](https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813 "2015-11-17T14:29:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vgm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vgm/32/5979_2.png) [@vgm](https://discuss.elastic.co/u/vgm)\
**Post date:** [November 17, 2015, 2:29pm UTC](https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813/1 "2015-11-17T14:29:37Z")

</div>

Hi all,

I'm wondering whether I can use logstash for ensuring accuracy during log collection (i.e. that no events will be skipped during input or output).

For instance, let's consider the following configuration:

```
input {
	file {
		type => "test"
		path => "/root/development/_in/*.log"
		start_position => "beginning"
	}
}

output {
	file {
		path => "./_out/test.log"
	}
}

```

Now, if I initiate logstash instance and then I copy-paste a 50k-log file in the "\_in" directory (or let a process generate it), I would expect to see all the 50k events/lines in the "test.log" output file. Instead, what I get is a varying-length output (i.e. I could be missing 2 or 3 or 10 or 20 or even none events).

I wonder if this has to do with the pipeline's throttling. So far I have tried the following without any luck:

- Running on both Windows and Linux environments
- Trying logstash versions 1.5.2, 1.5.5 and 2.0.0
- Increasing heap size to 1g
- Instead of writing output to file, sending them over the network with lumberjack

Please see below the structure of an exemplary input file.

![](https://us1.discourse-cdn.com/elastic/original/2X/3/3f3b30b5514e4cc6658701823a5238d1fb64ca22.PNG)

Thanks in advance for any reply.

V.

---

<div class="post-metadata">

**Author:** ![Jackal9301](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jackal9301/32/5748_2.png) [@Jackal9301](https://discuss.elastic.co/u/Jackal9301)\
**Post date:** [November 17, 2015, 5:35pm UTC](https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813/2 "2015-11-17T17:35:57Z")

</div>

I'm not sure if I have the write answer or your process but you may need a sincedb\_path:

```
input {
file {
	type => "test"
	path => "/root/development/_in/*.log"
	start_position => "beginning"
            sincedb_path => "PATHNAME"
}
}

```

If you are running and adding and rerunning the file you need to delete that sincedb file it creates everytime. Hopefully this helps!!!

---

<div class="post-metadata">

**Author:** ![vgm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vgm/32/5979_2.png) [@vgm](https://discuss.elastic.co/u/vgm)\
**Post date:** [November 18, 2015, 12:11pm UTC](https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813/3 "2015-11-18T12:11:25Z")

</div>

Hi Jackal,

Thanks for your reply.

Unfortunately, the reported issue is not with files or new lines that were not discovered, but I gave it a try anyway, without any luck: for instance I generated a new file, logstash identified it, but still some lines where (randomly) missing at the output, without any (obvious) reasons when I enabled debug.

Hopefully, this can be reproduced easily:

1. Lauch logstash with the aforementioned configuration
2. Execute the following command to generate input for logstash:  
`for i in {1..50000}; do echo "This, is, a, csv, and, this, is, line, no, $i"; done > _in/testme.log`
3. Wait until logstash picks up the file (I think this is configurable by the `stat interval` option)
4. Notice by counting the out put (e.g. `wc -l _out/test.log`) the result is not 50000 lines (e.g. it could be 49994 or any other randomly changing count on each run).

Thanks, V.

---

<div class="post-metadata">

**Author:** ![vgm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vgm/32/5979_2.png) [@vgm](https://discuss.elastic.co/u/vgm)\
**Post date:** [November 23, 2015, 9:46am UTC](https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813/4 "2015-11-23T09:46:19Z")

</div>

Hello,

The "issue" I encountered is actually occurring due to the caching feature of the pipeline.

By configuring the logstash `file` output plugin with the `flush_interval => 0` option flushing occurs for every message.

See more at: [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html#plugins-outputs-file-flush\_interval](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html#plugins-outputs-file-flush_interval)

Thanks, V.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:21am UTC](https://discuss.elastic.co/t/solved-accuracy-randomly-missing-events/34813/5 "2017-07-06T05:21:44Z")

</div>