Significance of @timestamp in Index Patterns

Elastic Stack Elasticsearch
1

Hi All,

What is the significance of @timestamp field that populates by default in all indices in Kibana?

image

I ask this question as this time differs with the recvdTime field that is set in the application and gets displayed as below:

image

At times the difference between the two is in hours.

Does that mean there is a delay or a choke in the network on within elasticsearch i.e. between filebeat and Logstash?

Thanks

2

recvdTime is not a standard field from beats or logstash afaik, and is likely coming from a field in your log or from a step in your processing pipeline.

@timestamp can be set by beats or logstash, for example with windows logs, the @timestamp field is set to the time from the log.

If there is no @timestamp field in the document when it arrives in Elasticsearch then Elasticsearch will set the timestamp to be the time it received the document

3

Thanks. Yes recvdTime is a field coming from the logs.

So to my question. If @timestamp is a field introduced by ES then does a difference in 6 to 7 hours between recvdTime and @timestamp indicate a slowness or buffering either in filebeat to Logstash communication or due to a network slowness?

4

If you have Logstash in your pipeline I believe it will set the @timestamp field to the timestamp it received it. There are several things that can cause differencs in timestamps, e.g.:

  • Differences in system time on the hosts that the components run on.
  • Timezone issues when parsing timestamps. This often results in a large and largely static difference.
  • Delays in reading the initial log.
  • Issues sending data down the pipeline, e.g. from Filebeat to Logstash or from any of these components to Elasticsearch. This can happen if some component goes down or is temporarily unavailable.

I would recommend analysing the logs and see if there is any pattern across the ones that show delay, e.g. a specific log format, source or host. That may help you narrow down where the issue is. Also look at logs from the various components to see if there are any issues reported.

5

Thanks. This issue is seen during performance test (sorry forgot to mention it earlier) when 10X or more load is applied to the application.

Logstash and Elasticsearch instances are hosted on the same servers.

Both the Filebeat and Logstash/ES servers have exactly the same date/ time.

I guess I need to analyze logs of different components at the time of the issue. It surely seems a load related issue.

6

If you are running a load test it is posible that some part of the chain is a bottleneck. If this was the case I would expect to see the delay grow over time. Maybe you can create an index pipeline that calculates the delay in seconds and stores this in a new field on the events. Then use this to update the data ingested as well as any new data. That way you can easily plot delay over time and slice by sources and/or components in Kibana to identify patterns. I believe this is described in this blog post as well as here.

7

Thanks. Also could the slowness be because we are using a 7.9.3 ELK cluster?

8

It could be because Elasticsearch is not able to keep up, but even though the version you are using is very old and has been EOL a long time (you should look to upgrade as soon as possible) I do not think the version in itself is the cause. I have benchmarked high ingest loads on significantly older versions.

When it comes to indexing it is often disk I/O that is the limiting factor. I would recommend you monitor disk I/O using e.g. iostat -x, on the Elasticsearch nodes to see of this looks like a possible cause. What type of storage are you using? Do you have monitoring installed so you can see the indexing rate you are achieving?

9

Thanks. One side question. What is the significance of "Time filter" when setting up an index pattern or Data view?

image

When setting it up it says Select a primary time field for use with the global time filter

We have our ELK hosted on cloud VMs using virtual disks and will monitor disk i/o