# ELK data not getting saved **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2018, 6:29am UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854 "2018-05-15T06:29:26Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Rahul\_Kumar2](https://avatars.discourse-cdn.com/v4/letter/r/977dab/32.png) [@Rahul\_Kumar2](https://discuss.elastic.co/u/Rahul_Kumar2)\ **Post date:** [May 15, 2018, 6:29am UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/1 "2018-05-15T06:29:26Z")
Hi awesome people We are using ELK with Amazon SQS in an in-house server. `Elasticsearch log` > [2018-05-15T11:39:03,411][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][982956][151886] duration [8.8s], collections [1]/[9.8s], total [8.8s]/[4.8d], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [399.4mb]-\>[288.8mb]/[399.4mb]}{[survivor] [41.6mb]-\>[0b]/[49.8mb]}{[old] [3.5gb]-\>[3.5gb]/[3.5gb]} > [2018-05-15T11:39:03,411][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982956] overhead, spent [8.8s] collecting in the last [9.8s] > [2018-05-15T11:39:13,309][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][982958][151887] duration [8s], collections [1]/[8.8s], total [8s]/[4.8d], memory [3.9gb]-\>[3.8gb]/[3.9gb], all\_pools {[young] [399.4mb]-\>[296.9mb]/[399.4mb]}{[survivor] [1.8mb]-\>[0b]/[49.8mb]}{[old] [3.5gb]-\>[3.5gb]/[3.5gb]} > [2018-05-15T11:39:13,309][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982958] overhead, spent [8s] collecting in the last [8.8s] > [2018-05-15T11:39:20,377][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982965] overhead, spent [432ms] collecting in the last [1s] > [2018-05-15T11:39:30,031][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982966] overhead, spent [9s] collecting in the last [9.6s] > [2018-05-15T11:39:39,355][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982975] overhead, spent [442ms] collecting in the last [1s] > [2018-05-15T11:39:47,357][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982983] overhead, spent [401ms] collecting in the last [1s] > [2018-05-15T11:39:56,612][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982984] overhead, spent [9s] collecting in the last [9.2s] > [2018-05-15T11:40:13,156][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][982992][151893] duration [9.2s], collections [1]/[9.3s], total [9.2s]/[4.8d], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [399.4mb]-\>[287mb]/[399.4mb]}{[survivor] [48.3mb]-\>[0b]/[49.8mb]}{[old] [3.5gb]-\>[3.5gb]/[3.5gb]} > [2018-05-15T11:40:13,156][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][982992] overhead, spent [9.2s] collecting in the last [9.3s] > [2018-05-15T11:40:22,385][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][983001] overhead, spent [432ms] collecting in the last [1s] > [2018-05-15T11:40:30,386][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][983009] overhead, spent [442ms] collecting in the last [1s] `Logstash log ` > [2018-05-15T09:22:03,420][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=\>"es\_rejected\_execution\_exception", "reason"=\>"rejected execution of org.elasticsearch.transport.TransportService$7@254974e8 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@7de20ac8[Running, pool size = 6, active threads = 6, queued tasks = 50, completed tasks = 60068]]"}) > [2018-05-15T09:22:03,420][ERROR][logstash.outputs.elasticsearch] Retrying individual actions > [2018-05-15T09:22:03,420][ERROR][logstash.outputs.elasticsearch] Action > [2018-05-15T09:22:03,420][ERROR][logstash.outputs.elasticsearch] Action After seeing the log and doing a lookup in the forum, for now, what I can understand is, memory allocated for various component of ELK is almost full. And for now, the solution which I can think of now is to free some space, but I am not sure what and how to clear space. `Note: We don't need last year data, that can be cleared from everywhere ` Need help, thanks. ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [May 16, 2018, 5:07pm UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/2 "2018-05-16T17:07:13Z")
> [@Rahul\_Kumar2](#): > > [gc][old][982956][151886] duration [8.8s], collections [1]/[9.8s], total [8.8s]/[4.8d], memory [3.9gb]-\>[3.7gb]/[3.9gb] @Rahul_Kumar2, what is the output of `_cluster/health?v`? Are you over sharded? My current presumption would be your cluster is under memory pressure from having too many indices for the current resources, thus you are seeing bulk rejections see: `_cat/thread_pool?v` ergo the 429 push back to logstash. Note the above excerpt of log tells us that the older generation of garbage collection is taking just less than 10seconds and was able free up .2GB of memory from the Elasticsearch heap. ---
**Author:** ![Rahul\_Kumar2](https://avatars.discourse-cdn.com/v4/letter/r/977dab/32.png) [@Rahul\_Kumar2](https://discuss.elastic.co/u/Rahul_Kumar2)\ **Post date:** [May 17, 2018, 7:21am UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/3 "2018-05-17T07:21:11Z")
@JKhondhu Thanks for the reply. I think I need to clear data (indices and logs). Should I clear the data? Will that help? I am not sure could you guide me on this? Meanwhile here the results you asked for `curl -XGET "localhost:9999/_cluster/health"` > { "cluster\_name":"xxxx-api-logs", > "status":"red", > "timed\_out":false, > "number\_of\_nodes":1, > "number\_of\_data\_nodes":1, > "active\_primary\_shards":3577, > "active\_shards":3577, > "relocating\_shards":0, > "initializing\_shards":4, > "unassigned\_shards":3971, > "delayed\_unassigned\_shards":0, > "number\_of\_pending\_tasks":144, > "number\_of\_in\_flight\_fetch":0, > "task\_max\_waiting\_in\_queue\_millis":67954, > "active\_shards\_percent\_as\_number":47.364936440677965} `curl -XGET "localhost:9999/_cat/thread_pool?v" ` > node\_name name active queue rejected > node-1 bulk 1 0 454 > node-1 fetch\_shard\_started 0 0 0 > node-1 fetch\_shard\_store 0 0 0 > node-1 flush 3 1 0 > node-1 force\_merge 0 0 0 > node-1 generic 1 0 0 > node-1 get 0 0 0 > node-1 index 0 0 0 > node-1 listener 0 0 0 > node-1 management 4 0 0 > node-1 refresh 0 0 0 > node-1 search 0 0 0 > node-1 snapshot 0 0 0 > node-1 warmer 0 0 0 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 17, 2018, 8:06am UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/4 "2018-05-17T08:06:50Z")
You probably have too many shards per node. May I suggest you look at the following resources about sizing: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing) > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... > **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)** > > Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser... ---
**Author:** ![Rahul\_Kumar2](https://avatars.discourse-cdn.com/v4/letter/r/977dab/32.png) [@Rahul\_Kumar2](https://discuss.elastic.co/u/Rahul_Kumar2)\ **Post date:** [May 17, 2018, 12:29pm UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/5 "2018-05-17T12:29:05Z")
@JKhondhu @JKhondhu Implemented `filebeat`, apparently everything started to work, but don't know how long it will one. Do you guys have any idea, let me know. Thanks ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [May 17, 2018, 12:49pm UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/6 "2018-05-17T12:49:04Z")
Hi, Implementing `filebeat` opposed to `logstash` will not solve your problem. You will see the same happen. You have too much data (indices) residing in Elasticsearch. You need to a. delete data past a retention period that you need or b. grow your cluster size. Close to 4000 shards on a single node with 4GB heap is crazy and asking for trouble. Please see all the links @dadoonet shared and start to put in place curator or your own job to delete indices past X amount of days and make resources available so your cluster is able to ingest new data coming in via logstash. ---
**Author:** ![Rahul\_Kumar2](https://avatars.discourse-cdn.com/v4/letter/r/977dab/32.png) [@Rahul\_Kumar2](https://discuss.elastic.co/u/Rahul_Kumar2)\ **Post date:** [May 18, 2018, 5:01am UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/7 "2018-05-18T05:01:47Z")
@JKhondhu @dadoonet.. Thanks for the advice. I am looking in to the shared resources while also learning the ELK stack. For now, I have increased the heap size to 8GB and everything is working now. Will also put in curator. Regards ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 15, 2018, 5:15am UTC](https://discuss.elastic.co/t/elk-data-not-getting-saved/131854/8 "2018-06-15T05:15:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.