So the thing is I have filebeat that ship the log file to logstash then to the elasticsearch cluster. And I created the data view the get the messages on the discover page. I wonder where this @timestamp field comes from? Is the time elasticsearch received the message? or the time logstash received the message? or filebeat scanned the message?
@timestamp is from LS, they explained how is generated.
I assume you are confused why you have time difference. If you like to update @timestamp to correct value-from a log, you have to parse the message, extract date from the begging, 2nd field is IP of something, etc.
So, you need grok or dissect to split the message in the fields.
After that convert 1. field to date format and overwrite @timestamp. This is how you will have @timestamp from the log file.
match => [ "timestamp", "MMM d HH:mm:ss.SSS", "MMM dd HH:mm:ss.SSS" ]
timezone => "Asia/Dubai" # optionally
target=> "@timestamp" # this is default
The reason I am doing this is I want to find a way to show the logs in the Kibana in order. So I am thinking it could help if the @timestamp comes from Filebeat since Filebeat reads the file line by line.
I wonder if the @timestamp comes from Filebeat, then the message shown on the Kibana should be ordered if I sort by @timestamp
So I have created a file named b.log and this is its content:
Yes, it could affect the performance, the processing will be done by 1 worker only and sequentially, not in parallel.
It really depends on your data, you would need to be able to have another field to sort on, for example, you could parse your message and put the value after connection reset on a field and sort using this field as well.
For elastic it makes no difference, it really depends on your data and what you want to do with it.