# Elasticsearch data indexing for logstash **URL:** \ **Category:** Elasticsearch\ **Created:** [November 19, 2015, 4:15pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066 "2015-11-19T16:15:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 19, 2015, 4:15pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/1 "2015-11-19T16:15:37Z")
As we know elasticsearch stores the logstash indices in the format logstash-yyyy.mm.dd. Does elastic search creates new indexes for new date by re-indexing previous day indexes? Ex: I am observing every day logstash-\* folders content in getting increased by twice the previous day size. logstash-2015.11.17 was ~500MB, logstash-2015.11.18 was ~1.5 GB and logstash-2015.11.19 is \> 3 GB ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 19, 2015, 4:22pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/2 "2015-11-19T16:22:36Z")
> As we know elasticsearch stores the logstash indices in the format logstash-yyyy.mm.dd. Does Elasticsearch creates new indexes for new date by re-indexing previous day indexes? No, Elasticsearch doesn't reindex data on its own. > Ex: I am observing every day logstash-\* folders content in getting increased by twice the previous day size. > logstash-2015.11.17 was ~500MB, > logstash-2015.11.18 was ~1.5 GB and > logstash-2015.11.19 is \> 3 GB And you're not just logging more data? ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 19, 2015, 4:53pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/3 "2015-11-19T16:53:20Z")
If not, how logstash folder is becoming twice the previous day's size? ---
**Author:** ![Drew\_Town](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drew_town/32/44829_2.png) [@Drew\_Town](https://discuss.elastic.co/u/Drew_Town)\ **Post date:** [November 19, 2015, 5:04pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/4 "2015-11-19T17:04:12Z")
Did you upgrade to 2.0? What is the document count on each index? Did your documents get a lot bigger? Did you turn on doc values? ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 19, 2015, 5:17pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/5 "2015-11-19T17:17:46Z")
Yes, i am using elasticsearch 2.0. and Yes every day my document's count is getting increased. ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 20, 2015, 3:12pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/6 "2015-11-20T15:12:48Z")
With respect to above query below is some more information/issues. 1. As per Marvel logstash-2015.11.19 data size is 1,004.8MB and Document Count is 2.4m logstash-2015.11.20 data size is 263.7MB and Document count is 618.5k 2. But I am not able to view any data related to logstash-2015.11.20 in Kibana. 3. Later investigated the log files and got to know last update to elasticsearch index happened on 2015-11-19 22:04:09,842 and last good contact between logstash and elasticsearch is at ~Thu Nov 19 23:00:00 CST 2015. 4. So while creating new indices folder by elasticsearch, logstash is loosing connectivity with elasticsearch. I have observed the same behaviour on 2015.11.17 but restarting logstash instance resolved the problem but triggered my initial query on this topic. My ELK stack setup flow Server1 -\> Server2 -\> Server3 Server1 -\> Logstash forwarder Server2 -\> Logstash lumberjack input plugin -\> Logstash Kafka output plugin Server3 -\> Logstash kafka input plugin -\> grok filter plugin -\> Logstash elasticsearch output plugin. This is the POC i am working and planning to implement production setup in next week. Any suggestions will be great help. ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 20, 2015, 6:32pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/7 "2015-11-20T18:32:56Z")
Some more information. After executing below steps, data started getting coming in Kibana. 1. Restarted logstash instance 2. Restated elasticsearch instance After above both steps, still not able to view the data in Kibana. Then in Kibana i have created new index pattern as logstash-2011.11.20 from settings -\>indices -\> create. As i mentioned in the topic [ElasticSearch indexing events from previous date into current date logstash-\* folder](https://discuss.elastic.co/t/elasticsearch-indexing-events-from-previous-date-into-current-date-logstash-folder/35177/1), i started seeing the data populating in Kibana. ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 25, 2015, 8:21pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/8 "2015-11-25T20:21:17Z")
Below is screen shot for index sizes for 2 days. ![](https://us1.discourse-cdn.com/elastic/original/2X/0/073c22a0a3b7e5cfb8cc0c0d9fc3542f956b5cbf.png) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 26, 2015, 7:00am UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/9 "2015-11-26T07:00:35Z")
At this point it's not clear to me what the problem is. You're populating ES and it sounds like you can see the data in Kibana. ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 30, 2015, 6:03pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/10 "2015-11-30T18:03:05Z")
Above comparison for one of the question in the same topic. Below the question asked to provide the data. What is the document count on each index? Did your documents get a lot bigger? Did you turn on doc values? Nope I am not seeing any data in kibana. I could see EC indexed the data from Nov-26 to Nov-29. But i could see the data till Nov-26. Again EC stopped indexing data from Nov-30. I am not sure, if i am missing any configuration settings. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:34pm UTC](https://discuss.elastic.co/t/elasticsearch-data-indexing-for-logstash/35066/11 "2017-07-05T23:34:47Z")