# Elastic search stops recording data after reaching a certain count **URL:** \ **Category:** Logstash\ **Created:** [March 14, 2018, 7:02am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847 "2018-03-14T07:02:05Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 14, 2018, 7:02am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/1 "2018-03-14T07:02:06Z")
In Elasticsearch data is added till a certain count and then it stops processing after that point. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2018, 7:36am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/2 "2018-03-14T07:36:25Z")
I suspect you will need to provide more details in order for anyone to be able to help you with this. ---
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 14, 2018, 7:56am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/3 "2018-03-14T07:56:18Z")
I'm opening logstash from java to write data into elasticsearch This is my conf file: input { file { path =\> "D:\logstash\logstash-6.1.0\bin\repoprolog.log" start\_position =\> "beginning" codec =\> multiline { pattern =\> "%{DATE}\_%{TIME} - " negate =\> true what =\> "previous" auto\_flush\_interval =\> 1 } sincedb\_path =\> "/dev/null" ignore\_older =\> 0 } } filter { grok { match =\> {"message" =\> "%{DATE}\_%{TIME} - %{POSINT:value} %{DATA:status} %{WORD:result} %{GREEDYDATA:final}"} } } output { elasticsearch { hosts =\> ["localhost:9200"] } stdout { codec =\> rubydebug } } this is my logstash log: [2018-03-14T13:10:58,562][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"D:/logstash/logstash-6.1.0/modules/fb\_apache/configuration"} [2018-03-14T13:10:58,577][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"D:/logstash/logstash-6.1.0/modules/netflow/configuration"} [2018-03-14T13:10:58,749][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-03-14T13:10:59,171][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.1.0"} [2018-03-14T13:10:59,619][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2018-03-14T13:11:03,103][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2018-03-14T13:11:03,103][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"} [2018-03-14T13:11:03,259][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [2018-03-14T13:11:03,306][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>nil} [2018-03-14T13:11:03,322][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} [2018-03-14T13:11:03,322][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "omit\_norms"=\>true}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}, "fields"=\>{"raw"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true, "ignore\_above"=\>256}}}}}, {"float\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"float", "mapping"=\>{"type"=\>"float", "doc\_values"=\>true}}}, {"double\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"double", "mapping"=\>{"type"=\>"double", "doc\_values"=\>true}}}, {"byte\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"byte", "mapping"=\>{"type"=\>"byte", "doc\_values"=\>true}}}, {"short\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"short", "mapping"=\>{"type"=\>"short", "doc\_values"=\>true}}}, {"integer\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"integer", "mapping"=\>{"type"=\>"integer", "doc\_values"=\>true}}}, {"long\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"long", "mapping"=\>{"type"=\>"long", "doc\_values"=\>true}}}, {"date\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"date", "mapping"=\>{"type"=\>"date", "doc\_values"=\>true}}}, {"geo\_point\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"geo\_point", "mapping"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "doc\_values"=\>true}, "@version"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true}, "geoip"=\>{"type"=\>"object", "dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip", "doc\_values"=\>true}, "location"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}, "latitude"=\>{"type"=\>"float", "doc\_values"=\>true}, "longitude"=\>{"type"=\>"float", "doc\_values"=\>true}}}}}}}} [2018-03-14T13:11:03,353][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [2018-03-14T13:11:03,447][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500, :thread=\>"#\"} pipeline is not started i guess and some data is stored in elasticsearch (in my case 1096 records are there but in elasticsearch only 16 are stored health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open logstash-2018.03.14 1 1 16 0 33.4kb 33.4kb ).plz help ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2018, 8:05am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/4 "2018-03-14T08:05:05Z")
> [@sunilkumar](#): > > sincedb\_path =\> "/dev/null" I believe this path should be set to "NUL" on Windows. ---
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 14, 2018, 8:28am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/5 "2018-03-14T08:28:19Z")
I've changed it . But this didn't solve my problem ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2018, 11:14am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/6 "2018-03-14T11:14:13Z")
> [@sunilkumar](#): > > ignore\_older =\> 0 Might this setting make Logstash ignore files? What happens if you remove this or set it to a larger value? ---
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 14, 2018, 11:28am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/7 "2018-03-14T11:28:51Z")
I tried removing this ,that didn't solve the problem. I tried setting ignore\_older =\> 10 i didn't get any response in elasticsearch ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2018, 11:48am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/8 "2018-03-14T11:48:35Z")
What is t he age of the files you are trying to read? Would 10 seconds be an appropriate value to catch them? ---
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 14, 2018, 11:55am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/9 "2018-03-14T11:55:17Z")
The problem is with logstash i guess . As you can see the logstash log file it stops at starting pipeline step i.e pipeline is not created. **The next step is continued after some delay like 5 minutes . I don't think the issue is with sinceDB** ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2018, 12:01pm UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/10 "2018-03-14T12:01:55Z")
Are there any errors reported in the Elasticsearch logs? ---
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 14, 2018, 12:11pm UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/11 "2018-03-14T12:11:18Z")
No errors in elasticsearch. I tried writing the data into a file instead of es .There also only partial data was written ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2018, 12:30pm UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/12 "2018-03-14T12:30:27Z")
Enable debug logging in Logstash and see if that provides any additional information. ---
**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Post date:** [March 15, 2018, 8:17am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/13 "2018-03-15T08:17:04Z")
Tried debugging but that didn't help much. Should i make any changes with jvm.options file considering the fact that i'm getting this issue only when i try to run from java ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2018, 8:17am UTC](https://discuss.elastic.co/t/elastic-search-stops-recording-data-after-reaching-a-certain-count/123847/14 "2018-04-12T08:17:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.