Timestamp of When ElasticSearch Receives Data?

Elastic Stack Elasticsearch
1

Hi ELK Gurus,

I have a pipeline of Docker containers, ending in an ELK stack of Logstash/Elasticsearch/Kibana. (ELK versions are 7.4.0, I know I need to upgrade.) Data flows into the pipeline from an external source, then is passed along all five containers until it is visible in Kibana. Everything works great and I’m very happy with ELK.

But recently my boss asked me if we could enhance the data by having Logstash run some DNS lookups and/or consult with an external database. My immediate worry is the end-to-end processing delay this would incur for my data records. I’d like to tell him, “Right now, it takes .1 ms (or whatever) between the time the data first enters the pipeline and when that data is visible in Kibana. If I add in a DNS/database lookup, that will add ~200 ms per record.”

My raw data schema includes an initial timestamp. Is there any way to get the timestamp of when Elasticsearch or Kibana first sees the same data? That way, I could take the delta between the two and time out how long it takes a data record to traverse my ELK pipeline. Thank you.

2

If you are using ingest processors, you can access the ingest metadata, which contains a timestamp of the retrieval of the document by Elasticsearch.

See https://www.elastic.co/guide/en/elasticsearch/reference/7.6/accessing-data-in-pipelines.html#accessing-ingest-metadata

3

Wow, thanks spinscale! Appreciate the tip

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.