# Infinite loop in ES, possible bug, please comment

**URL:** https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917
**Category:** Elasticsearch
**Created:** [January 11, 2016, 8:31pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917 "2016-01-11T20:31:50Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![Christopher\_Curzon](https://avatars.discourse-cdn.com/v4/letter/c/2acd7d/32.png) [@Christopher\_Curzon](https://discuss.elastic.co/u/Christopher_Curzon)
#### Post date: [January 11, 2016, 8:31pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/1 "2016-01-11T20:31:50Z")

</div>

So I'm using Logstash to load a csv file having 4998079 records in it. As it is loading I'm tracking the count in Elasticsearch and I notice that the count goes well beyond 5M and keeps climbing.

Then I notice that the console shows the following message:

A plugin had an unrecoverable error. Will restart this plugin.

```
  Plugin: <LogStash::Inputs::File path=>["/home/zed/logstash-2.1.0/source_data/nga/AIS_Ships_PD-3874.csv"], sincedb_path=>"/home/zed/logstash-2.1.0/sincedb_path/AIS_Ships_PD-3874.since", type=>"core2", start_position=>"beginning", codec=><LogStash::Codecs::Plain charset=>"UTF-8">, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">
  Error: No such file or directory - /home/zed/logstash-2.1.0/sincedb_path/AIS_Ships_PD-3874.since.14300.17459.522421 {:level=>:error}

```

Which explains that the load process has restarted from the beginning of the file, and that is why the record count continues to climb.

So out of curiosity, I wait to see if it will halt at the end of the second load. But ES issues the same fatal error message, and then restarts the data load for the third time. So I killed the logstash process.

The problem, which is clear from the error message, is that the sincedb\_path value specified a path which did not exist.

The consequence is that this sets up an infinite loop. Not good.

My thought is that ES should check path and permissions to verify that the sincedb file can be created _BEFORE_ starting the load.

Is there any reason why the file plugin ES (or Logstash) does not do this?

-- Chris curzon

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 12, 2016, 2:57am UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/2 "2016-01-12T02:57:59Z")

</div>

That's an LS error, not an ES one though?

---

<div class="post-metadata">

### Author: ![Christopher\_Curzon](https://avatars.discourse-cdn.com/v4/letter/c/2acd7d/32.png) [@Christopher\_Curzon](https://discuss.elastic.co/u/Christopher_Curzon)
#### Post date: [January 12, 2016, 8:33pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/3 "2016-01-12T20:33:12Z")

</div>

Yes. I would agree with you. It's a logstash error.

So, do you think this would be an error in the file plug-in? Or is it a feature based on some kind of optimization question.

An honest question. My experience suggests that, when one starts working with a system as complex and configurable as LS or ES, one has to decide how much trust to place in the system defaults. And what that usually means is that system defaults are not necessarily bugs, but instead are optimizations for a different use case. (So is there a use case for LS _not_ to check the validity of the sincedb\_path value?)

It's sort of like in a relational database the decision of whether or not to have an index. If you want all or most of the records in a table, then you don't want to use the index, but if you're fetching one a just a few records, then you do. And in Oracle or any other relational database, the optimizer usually make a decent decision on whether to use, or not.

But in LS or ES, there is no optimizer. (Is there?) So in the back of my mind is the possibility that I will--fairly easily--find some dark corner where something untoward happens.

Your thoughts?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 12, 2016, 9:38pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/4 "2016-01-12T21:38:15Z")

</div>

> [@Christopher\_Curzon](#):
>
> Error: No such file or directory - /home/zed/logstash-2.1.0/sincedb\_path/AIS\_Ships\_PD-3874.since.14300.17459.522421

Why does this happen? Does the file get created at all? Does the directory exist? Does the LS user have the permissions to write to this directory?

---

<div class="post-metadata">

### Author: ![Christopher\_Curzon](https://avatars.discourse-cdn.com/v4/letter/c/2acd7d/32.png) [@Christopher\_Curzon](https://discuss.elastic.co/u/Christopher_Curzon)
#### Post date: [January 12, 2016, 10:12pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/5 "2016-01-12T22:12:00Z")

</div>

The failure was because the directory didn't exist.

The file system had "sample\_data" in the path.

But the config file had "sample\_date" specifie as the path to use.

So the directory required by the config file did not exist.

I'm just wondering why LS did not check this fact before starting the processing.

---

<div class="post-metadata">

### Author: ![Christopher\_Curzon](https://avatars.discourse-cdn.com/v4/letter/c/2acd7d/32.png) [@Christopher\_Curzon](https://discuss.elastic.co/u/Christopher_Curzon)
#### Post date: [January 12, 2016, 10:18pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/6 "2016-01-12T22:18:20Z")

</div>

I just did a test. In the config file I specified

input {  
file {  
path =\> "/home/zed/logstash-2.1.0/source\_data/gdelt/20151124.export.CSV"  
sincedb\_path =\> "/homex/zed/logstash-2.1.0/load\_conf/gdelt.since"  
type =\> "gdelt"  
start\_position =\> "beginning"  
}  
}

notice I gave /homex/zed.... as the sincedb\_path value. This path doesn't exist.

But configtest, says this configuration is OK

$ bin/logstash -f load\_conf/gdelt01.conf --configtest

Configuration OK

So I have to be very careful in this regard.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 12, 2016, 10:27pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/7 "2016-01-12T22:27:00Z")

</div>

Feel free to raise an issue on the file input repo on this, it does seem this should be checked.

---

<div class="post-metadata">

### Author: ![Christopher\_Curzon](https://avatars.discourse-cdn.com/v4/letter/c/2acd7d/32.png) [@Christopher\_Curzon](https://discuss.elastic.co/u/Christopher_Curzon)
#### Post date: [January 13, 2016, 5:51pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/8 "2016-01-13T17:51:09Z")

</div>

I will do as you suggest.

As a newbie, can you tell me how? Or are there instructions for me to follow?

Thanks.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 13, 2016, 9:14pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/9 "2016-01-13T21:14:46Z")

</div>

Sure, just create a new issue here - [https://github.com/logstash-plugins/logstash-input-file/issues](https://github.com/logstash-plugins/logstash-input-file/issues)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:24pm UTC](https://discuss.elastic.co/t/infinite-loop-in-es-possible-bug-please-comment/38917/10 "2017-07-05T23:24:49Z")

</div>


