Slow ingestion speeds

Elastic Stack Elasticsearch
1

Good Morning all,

I have noticed that i appear to be having some ingestion speed issues... in a nut shell, if i look at historical data from a couple of hours ago and keep refreshing my screen, the amount of records increases when i am refreshing. Now to me that seems that Elastic is not ingesting the data at the point it is available.

Are there any ways in which one i can monitor this and two improve the speed of my ingestion.

If there is any further information i can provide, please let me know and i will be happy to share.

Thank you in advance.

2

Can you elaborate more on what you mean by this and where you are doing the refreshing?

3

I can - So i am setting a specific time (historical) in Kibana (using the Discover tab) and every time i am refreshing my search the records are increasing. This is still something happening currently.
Its worth noting i am using Elastic Cloud.
My HOT storage Node (i believe the one that is doing the ingestion) has 240GB storage and 8GB RAM.

4

How are you ingesting the data? Has the process finished when you are doing these refreshes?

5

What does CPU usage and CPU credits look like for that node?

6

I am ingesting the data using Fleet with the Elastic Agent installed onto my servers. The process has finished uploading, its just taking time to appear in Kibana.

7

My CPU credits and CPU usage do look pretty high - I am not sure how to read this. But CPU credits sometimes hit around 21k (not sure what that means) and my usage runs generally at 200% but i can see spikes to 800%.

8

Then it is possible that you are saturating the cluster which is why it is not able to keep up. It might be worth increasing the size of the nodes to see if that resolves the issue you are seeing.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.