# Elastic Unstable

**URL:** <https://discuss.elastic.co/t/elastic-unstable/350593>\
**Category:** Elasticsearch\
**Created:** [January 8, 2024, 2:25pm UTC](https://discuss.elastic.co/t/elastic-unstable/350593 "2024-01-08T14:25:34Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 8, 2024, 2:25pm UTC](https://discuss.elastic.co/t/elastic-unstable/350593/1 "2024-01-08T14:25:34Z")

</div>

Hi Team Elastic,

I have been stressful latelty because my logs are coming to Elasticsearch delay for about 10 hours.

I have 3 nodes,  
Node 1: master, ingest, transform, resource: 16vCPU, 16GB, 500GB  
Node 2: data\_hot, resource: 16vCPU, 64GB, 2 TB  
Node 3: data\_, data\_warm : 16vCPU, 64TB, 10TB

Please help, I have been troubleshooting for 1 month and have no result

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [January 8, 2024, 6:06pm UTC](https://discuss.elastic.co/t/elastic-unstable/350593/2 "2024-01-08T18:06:08Z")

</div>

Hi @Dea_Agra,

Welcome back! Which version of Elasticsearch are you using? Can you give us more information on the troubleshooting you've done. For example have you checked the output of the [`_cluster/_health` API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 8, 2024, 6:37pm UTC](https://discuss.elastic.co/t/elastic-unstable/350593/3 "2024-01-08T18:37:31Z")

</div>

What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/cluster-stats.html)?

What type of hardware is the cluster deployed on? What type of storage are you using?

How much data are you indexing per day (or should the cluster be indexing if it was keeping up)?

What are you using to index data into the cluster?

Do you have monitoring installed?

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 9, 2024, 10:33am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/4 "2024-01-09T10:33:00Z")

</div>

Continuing the discussion from [Elastic Unstable](https://discuss.elastic.co/t/elastic-unstable/350593/3):

> {  
> "\_nodes" : {  
> "total" : 3,  
> "successful" : 3,  
> "failed" : 0  
> },  
> "cluster\_name" : "xforce-cluster",  
> "cluster\_uuid" : "aspcDuHYQ6qRfE8flD5RXQ",  
> "timestamp" : 1704796161773,  
> "status" : "green",  
> "indices" : {  
> "count" : 130,  
> "shards" : {  
> "total" : 164,  
> "primaries" : 130,  
> "replication" : 0.26153846153846155,  
> "index" : {  
> "shards" : {  
> "min" : 1,  
> "max" : 2,  
> "avg" : 1.2615384615384615  
> },  
> "primaries" : {  
> "min" : 1,  
> "max" : 1,  
> "avg" : 1.0  
> },  
> "replication" : {  
> "min" : 0.0,  
> "max" : 1.0,  
> "avg" : 0.26153846153846155  
> }  
> }  
> },  
> "docs" : {  
> "count" : 1386300702,  
> "deleted" : 1490912  
> },  
> "store" : {  
> "size\_in\_bytes" : 1845130506044,  
> "total\_data\_set\_size\_in\_bytes" : 1845130506044,  
> "reserved\_in\_bytes" : 0  
> },  
> "fielddata" : {  
> "memory\_size\_in\_bytes" : 102296,  
> "evictions" : 0  
> },  
> "query\_cache" : {  
> "memory\_size\_in\_bytes" : 3533173,  
> "total\_count" : 17121836,  
> "hit\_count" : 4152,  
> "miss\_count" : 17117684,  
> "cache\_size" : 614,  
> "cache\_count" : 614,  
> "evictions" : 0  
> },  
> "completion" : {  
> "size\_in\_bytes" : 0  
> },  
> "segments" : {  
> "count" : 2272,  
> "memory\_in\_bytes" : 265190208,  
> "terms\_memory\_in\_bytes" : 213271552,  
> "stored\_fields\_memory\_in\_bytes" : 5241472,  
> "term\_vectors\_memory\_in\_bytes" : 0,  
> "norms\_memory\_in\_bytes" : 26851968,  
> "points\_memory\_in\_bytes" : 0,  
> "doc\_values\_memory\_in\_bytes" : 19825216,  
> "index\_writer\_memory\_in\_bytes" : 21342876,  
> "version\_map\_memory\_in\_bytes" : 5526,  
> "fixed\_bit\_set\_memory\_in\_bytes" : 63466048,  
> "max\_unsafe\_auto\_id\_timestamp" : 1704795307981,  
> "file\_sizes" : { }  
> },  
> "mappings" : {  
> "field\_types" : [  
> {  
> "name" : "alias",  
> "count" : 920,  
> "index\_count" : 32,  
> "script\_count" : 0  
> },  
> {  
> "name" : "binary",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "boolean",  
> "count" : 2921,  
> "index\_count" : 94,  
> "script\_count" : 0  
> },  
> {  
> "name" : "byte",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "constant\_keyword",  
> "count" : 10,  
> "index\_count" : 4,  
> "script\_count" : 0  
> },  
> {  
> "name" : "date",  
> "count" : 4229,  
> "index\_count" : 110,  
> "script\_count" : 0  
> },  
> {  
> "name" : "date\_nanos",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "date\_range",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "double",  
> "count" : 823,  
> "index\_count" : 30,  
> "script\_count" : 0  
> },  
> {  
> "name" : "double\_range",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "flattened",  
> "count" : 289,  
> "index\_count" : 23,  
> "script\_count" : 0  
> },  
> {  
> "name" : "float",  
> "count" : 815,  
> "index\_count" : 48,  
> "script\_count" : 0  
> },  
> {  
> "name" : "float\_range",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "geo\_point",  
> "count" : 222,  
> "index\_count" : 32,  
> "script\_count" : 0  
> },  
> {  
> "name" : "geo\_shape",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "half\_float",  
> "count" : 57,  
> "index\_count" : 15,  
> "script\_count" : 0  
> },  
> {  
> "name" : "integer",  
> "count" : 206,  
> "index\_count" : 32,  
> "script\_count" : 0  
> },  
> {  
> "name" : "integer\_range",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "ip",  
> "count" : 2879,  
> "index\_count" : 36,  
> "script\_count" : 0  
> },  
> {  
> "name" : "ip\_range",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "keyword",  
> "count" : 118612,  
> "index\_count" : 110,  
> "script\_count" : 0  
> },  
> {  
> "name" : "long",  
> "count" : 28756,  
> "index\_count" : 97,  
> "script\_count" : 0  
> },  
> {  
> "name" : "long\_range",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "nested",  
> "count" : 105,  
> "index\_count" : 33,  
> "script\_count" : 0  
> },  
> {  
> "name" : "object",  
> "count" : 22071,  
> "index\_count" : 109,  
> "script\_count" : 0  
> },  
> {  
> "name" : "scaled\_float",  
> "count" : 2,  
> "index\_count" : 2,  
> "script\_count" : 0  
> },  
> {  
> "name" : "shape",  
> "count" : 1,  
> "index\_count" : 1,  
> "script\_count" : 0  
> },  
> {  
> "name" : "short",  
> "count" : 2122,  
> "index\_count" : 22,  
> "script\_count" : 0  
> },  
> {  
> "name" : "text",  
> "count" : 28353,  
> "index\_count" : 102,  
> "script\_count" : 0  
> },  
> {  
> "name" : "version",  
> "count" : 3,  
> "index\_count" : 3,  
> "script\_count" : 0  
> }  
> ],  
> "runtime\_field\_types" :   
> },  
> "analysis" : {  
> "char\_filter\_types" : ,  
> "tokenizer\_types" : ,  
> "filter\_types" : ,  
> "analyzer\_types" : ,  
> "built\_in\_char\_filters" : ,  
> "built\_in\_tokenizers" : ,  
> "built\_in\_filters" : ,  
> "built\_in\_analyzers" :   
> },  
> "versions" : [  
> {  
> "version" : "7.17.12",  
> "index\_count" : 130,  
> "primary\_shard\_count" : 130,  
> "total\_primary\_bytes" : 1839085416891  
> }  
> ]  
> },  
> "nodes" : {  
> "count" : {  
> "total" : 3,  
> "coordinating\_only" : 0,  
> "data" : 3,  
> "data\_cold" : 0,  
> "data\_content" : 0,  
> "data\_frozen" : 0,  
> "data\_hot" : 1,  
> "data\_warm" : 1,  
> "ingest" : 3,  
> "master" : 1,  
> "ml" : 0,  
> "remote\_cluster\_client" : 1,  
> "transform" : 1,  
> "voting\_only" : 0  
> },  
> "versions" : [  
> "7.17.12"  
> ],  
> "os" : {  
> "available\_processors" : 48,  
> "allocated\_processors" : 48,  
> "names" : [  
> {  
> "name" : "Linux",  
> "count" : 3  
> }  
> ],  
> "pretty\_names" : [  
> {  
> "pretty\_name" : "Ubuntu 22.04.3 LTS",  
> "count" : 3  
> }  
> ],  
> "architectures" : [  
> {  
> "arch" : "amd64",  
> "count" : 3  
> }  
> ],  
> "mem" : {  
> "total\_in\_bytes" : 151832961024,  
> "free\_in\_bytes" : 9252524032,  
> "used\_in\_bytes" : 142580436992,  
> "free\_percent" : 6,  
> "used\_percent" : 94  
> }  
> },  
> "process" : {  
> "cpu" : {  
> "percent" : 65  
> },  
> "open\_file\_descriptors" : {  
> "min" : 509,  
> "max" : 1543,  
> "avg" : 1105  
> }  
> },  
> "jvm" : {  
> "max\_uptime\_in\_millis" : 322513325,  
> "versions" : [  
> {  
> "version" : "20.0.2",  
> "vm\_name" : "OpenJDK 64-Bit Server VM",  
> "vm\_version" : "20.0.2+9-78",  
> "vm\_vendor" : "Oracle Corporation",  
> "bundled\_jdk" : true,  
> "using\_bundled\_jdk" : true,  
> "count" : 3  
> }  
> ],  
> "mem" : {  
> "heap\_used\_in\_bytes" : 42634670272,  
> "heap\_max\_in\_bytes" : 73014444032  
> },  
> "threads" : 400  
> },  
> "fs" : {  
> "total\_in\_bytes" : 13591541923840,  
> "free\_in\_bytes" : 11706572861440,  
> "available\_in\_bytes" : 11020294971392  
> },  
> "plugins" : ,  
> "network\_types" : {  
> "transport\_types" : {  
> "security4" : 3  
> },  
> "http\_types" : {  
> "security4" : 3  
> }  
> },  
> "discovery\_types" : {  
> "zen" : 3  
> },  
> "packaging\_types" : [  
> {  
> "flavor" : "default",  
> "type" : "deb",  
> "count" : 3  
> }  
> ],  
> "ingest" : {  
> "number\_of\_pipelines" : 55,  
> "processor\_stats" : {  
> "append" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "conditional" : {  
> "count" : 1366046566,  
> "failed" : 6983433,  
> "current" : 7,  
> "time\_in\_millis" : 454680900  
> },  
> "convert" : {  
> "count" : 112513696,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 1622979  
> },  
> "csv" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "date" : {  
> "count" : 224609970,  
> "failed" : 224609970,  
> "current" : 0,  
> "time\_in\_millis" : 6425730  
> },  
> "foreach" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "geoip" : {  
> "count" : 226514709,  
> "failed" : 502100,  
> "current" : 0,  
> "time\_in\_millis" : 5775351  
> },  
> "grok" : {  
> "count" : 16412993,  
> "failed" : 3369972,  
> "current" : 7,  
> "time\_in\_millis" : 1457124475  
> },  
> "gsub" : {  
> "count" : 13702480,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 109622  
> },  
> "join" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "kv" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "lowercase" : {  
> "count" : 449219942,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 7622456  
> },  
> "remove" : {  
> "count" : 224818681,  
> "failed" : 112304985,  
> "current" : 0,  
> "time\_in\_millis" : 2601647  
> },  
> "rename" : {  
> "count" : 4063909737,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 2750237  
> },  
> "script" : {  
> "count" : 450013516,  
> "failed" : 112305172,  
> "current" : 0,  
> "time\_in\_millis" : 11491545  
> },  
> "set" : {  
> "count" : 487857540,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 5728724  
> },  
> "set\_security\_user" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "split" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "uppercase" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> },  
> "uri\_parts" : {  
> "count" : 208711,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 2203  
> },  
> "user\_agent" : {  
> "count" : 0,  
> "failed" : 0,  
> "current" : 0,  
> "time\_in\_millis" : 0  
> }  
> }  
> }  
> }  
> }

this is the result of the cluster stats

1. I am using SAS HDD

2. it’s about 29.000.000 data /per daya

3. Logstash

4. Yes i am using monitoring installed

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 9, 2024, 2:20pm UTC](https://discuss.elastic.co/t/elastic-unstable/350593/5 "2024-01-09T14:20:50Z")

</div>

hi is there any update? please help😭

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 9, 2024, 2:38pm UTC](https://discuss.elastic.co/t/elastic-unstable/350593/6 "2024-01-09T14:38:51Z")

</div>

> [@Dea\_Agra](#):
>
> I am using SAS HDD

What does I/O statistics, e.g. await and disk utilisation, look like on the nodes? You can get this through e.g. `iostat -x` if you are running Linux.

> [@Dea\_Agra](#):
>
> Yes i am using monitoring installed

Do you see any high CPU usage or heap usage? Do you see anything in the Elasticsearch logs around long or frequent garbage collection on any of the nodes?

> [@Dea\_Agra](#):
>
> I have been stressful latelty because my logs are coming to Elasticsearch delay for about 10 hours.

How have you determined this? That is a huge lag. Are you setting timestamp fields properly? Timestamps must be in UTC and I have seen many users not account for this and therefore insert data that is futuredated as far as Elasticsearch is converned. This can result in it only showing up in Kibana once the timestamp is no longer in the future.

Which timezone are you in?

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 10, 2024, 10:17am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/7 "2024-01-10T10:17:52Z")

</div>

Hi, I found it because the ingest pipeline, do u have any idea so the logs won't delay if i am using the ingest pipeline?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 10, 2024, 11:21am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/8 "2024-01-10T11:21:37Z")

</div>

> [@Dea\_Agra](#):
>
> Hi, I found it because the ingest pipeline

How did you identify this?

Ingest pipelines are often limited by CPU. If you are seeing high CPU usage on your ingest nodes you may need to increase this. If you do not see high CPU, you may want to ingest data into Elasticsearch with a higher level of parallelism.

Another way to address this would be to make your ingest pipeline(s) more efficient. It looks like your `grok` and `conditional` filters are taking up most processing time so it may be worth starting there.

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 10, 2024, 11:33am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/9 "2024-01-10T11:33:21Z")

</div>

I delete the ingest pipeline configuration in index template and the elasticsearch could write until 15.000/s before is only max 2000/s and average 500/s.

What if I am using Logstash pipeline?

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 10, 2024, 11:35am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/10 "2024-01-10T11:35:12Z")

</div>

How about script painless? does it need high performance to process that?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 10, 2024, 11:35am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/11 "2024-01-10T11:35:19Z")

</div>

How are you sending data to Elasticsearch? What is your batch size? How many concurrent connections are you sending data over?

Do you see high CPU on your ingest node?

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 11, 2024, 4:12am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/12 "2024-01-11T04:12:29Z")

</div>

i send elasticsearch using Logstash. I attach my Logstash configuration:

> batch.size: 3000  
> batch.delay: 100  
> queue.type: memory  
> queue.max\_bytes: 2048mb  
> queue.checkpoint.writes: 4096

and this is my CPU usage

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d36b1089c0d693d5f09b70b8be5856c9faa0b3e3.jpeg)

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 11, 2024, 4:16am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/13 "2024-01-11T04:16:06Z")

</div>

I have three nodes of elasticsearch and I put ingest node in every node

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 11, 2024, 6:52am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/14 "2024-01-11T06:52:36Z")

</div>

How many CPU cores does the Logstash node(s) have?

> [@Dea\_Agra](#):
>
> batch.size: 3000

This is a quite large batch size. It would be interesting to see what the effect of decreasing this to the default value (or maybe 500) would have. Can you comment this out/change the value and restart Logstash?

> [@Dea\_Agra](#):
>
> What if I am using Logstash pipeline?

You can move the processing to Logstash, but be aware that it may require additional CPU resources. If your pipelines are very inefficient, this may not help much though.

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 11, 2024, 8:15am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/16 "2024-01-11T08:15:14Z")

</div>

What should I do to make the logs in real-time but still using the ingest pipeline? It takes time to convert the ingest pipeline to Logstash pipeline and also we can see from the CPU usage I sent before, I think it still enough for the CPU to processing the Logs but I don’t know why elasticsearch didn’t use the CPU

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 11, 2024, 8:26am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/17 "2024-01-11T08:26:19Z")

</div>

> [@Dea\_Agra](#):
>
> What should I do to make the logs in real-time but still using the ingest pipeline?

This is why I asked about the number of CPU cores allocated to Logstash. If I recall correctly the number of processing threads is set depending on the number of CPU cores available, and this will determine the level of processing parallelism, which limits how many concurrent connections to Elasticsearch that will be used.

Given that you have quite large batch sizes, it is possible you are sending relatively few batches to Elasticsearch concurrently. I believe batches are processed in a single thread, and if this is the case it could limit the amount of CPU used.

I would like to try increasing the number of threads Logstash uses and also decrease the batch size so multiple smaller loads are sent to Elasticsearch in parallel, thereby increasing CPU usage. Given the extensive processing done in ingest pipelines it is quite likely that each bulk request takes a long time to process.

Once we know what impact this have (if any) we can look at further potential improvements.

---

<div class="post-metadata">

**Author:** ![Dea\_Agra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dea_agra/32/49676_2.png) [@Dea\_Agra](https://discuss.elastic.co/u/Dea_Agra)\
**Post date:** [January 11, 2024, 10:38am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/18 "2024-01-11T10:38:34Z")

</div>

For Logstash, I am using 8vCPU cores. I would try to set the batch size as you recommended

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2024, 10:38am UTC](https://discuss.elastic.co/t/elastic-unstable/350593/19 "2024-02-08T10:38:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.