The timestamp is wrong

Elastic Stack Kibana
1

I found that the data write into ES doesn't match my real time .
it's later about 10 minutes.
this is my logsatsh config:

input{
	udp{
		host => "120.127.160.91"
		port => 5556
		codec => netflow	
	}
}
	filter{
		
	}
output{
elasticsearch {
                hosts => ["120.127.160.91:9200"]
				index => "netflow-%{year}-%{month}-%{day}"
			}
	
		stdout{codec=> rubydebug}
	
}

view data in kibana:

image
image.png1034×728 52.6 KB

2

Kibana simply shows the response from Elasticsearch, so any issue you have is going to be in your ingest pipeline somewhere. I'm not familiar with netflow myself, are you sure it's making it to Logstash in realtime, or is it possible that the data coming from the source is being cached or delayed perhaps? If you know the data is coming in quickly, perhaps something is misconfigured with your Elasticsearch node that is causing writes to take a long time to happen?

3

I am sure it's realtime because when I download a file from my pc, the flow comes to ES right away.
But the timestamp is later about 10 min to "now".

4

Oh, I think I misunderstood before, it's not that the data is taking 10 minutes to show up, it's that the timestamp is off by 10 minutes... is that right?

Are you collecting data and doing the processing on different machines? Maybe the clock on one of them is off?

5

yes, that's right. Sorry about my poor English :frowning:
I collect netflow on A machine and send them to B machine to process and ingest into ES,
but I think the time should match my B machine clock? and it's not related to A machine?

6

I think the time should match my B machine clock?

I'm honestly not sure. I would expect that the documents, being created on machine B, would have the timestamp already generated with machine B's clock, before it's sent off to machine A. But that's just my assumption, I have no idea if that's true. If the clocks are off though, especially off by 10 minutes, then you have the answer. Either way, it's worth syncing the clocks up as much as possible.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.