Staggered Logging in ELK Stack

Elastic Stack Elasticsearch
1

As of right now, I am logging into Kibana to check IIS logs that get forwarded and I am noticing that the last log being reported was from two hours ago. There are new ones showing up but it seems to be staggered by a couple of hours.

Could someone please point me in the right direction as to why this staggering would occur? Is this related to not enough resources, such as CPU and MEM? Disk related (not enough IOPS)? Where to begin?

Thanks!

2

Everything in ES and LS assumes UTC, KB takes that UTC timestamp and then alters it to match your browser timezone.

So I'd start there.

3

You missed what I asked completely. I am aware of the timezones that are being used. I asked about staggered logging.

4

There is no such thing, it either processes or it doesn't.

5

I don't think you have seen the level of logging I am mentioning. We are sending IIS, MS SQL, SharePoint ULS, PaaS Services and more. I am often looking at Kibana and seeing thirty minute+ delays to logs showing up. I can open up IIS logs and find logs within the last hour that aren't in Kibana yet, but they will be in an hour or two. It is STAGGERED. If you haven't worked with this much or experienced this issue, maybe you should monitor the thread since you might learn something.

6

Righto boss, good luck :slight_smile:

7

Thanks. I see you work for the Elastic team. I am surprised that as a 'Solutions Architect' you were not able to provide any insight besides 'it processes or it doesn't'. That's kind of a shame, especially since Elastic has been trying to get us to sign up for support. Real motivation to do that in this thread :wink:

8

What is the specification of the hardware your cluster and ingestion pipeline is running on? How much data are you trying to ingest per day?

There are a number of factors that can limit performance and cause the ingestion pipeline to not be able to keep up with the flow of data. Indexing into Elasticsearch can be quite CPU intensive and also cause high disk I/O load due to the constant merging of segments that take place. Make sure that you have followed the guidelines described here and then monitor your cluster to see if there is any apparent bottlenecks. If there is nothing that stands out, try adding additional load to see if there is spare capacity. If Elasticsearch is the bottleneck, scale up or out depending on what you current cluster looks like.

If Elasticsearch has spare indexing capacity, it may be the ingest pipeline that is limiting throughput, as these type of components can be very CPU intensive. Monitor the components of the ingest pipeline and tune it if possible. Try scaling up or out depending on what your pipeline looks like.