We are trying to get the range of logs between two timestamps. The @timestamp value is used to retreive the data.The issue is when two logs having the @timestamp value the logs are not sorted correctly. Please sugges tif there any other alternative for the same.
If the two records come from the same file and something indicating the order, e.g. file offset or line number, is available in the record, it may be possible to do a multilevel sort.
Do you know how to pass the line number of file to elastic search from logstash.
That will depend on which input mechanism you use. Filebeat is able to include the file offset, but is seems the Logstash file input plugin does not yet support this.
Currently the logs are ready locally from a path . Planning to integrate it with filebeat later. Okay then we'll check after integration with filebeat and revert on this . Thanks for the information.
i've integrated with filebeat and based on @timestamp and offset we are sorting the data. But the issue is with the same in the same @timestamp time (in millisecond) , it is processing more number of lines.
For example in file 1 if there are 30 lines processed in same time stamp and then continue processing the next 10 lines in the next file. Then the offset value in the next file will be lesser then the data in the 1st file. So there would be issue in sorting the data with offset and timestamp. Could you please suggest a solution for the same.
Regarding offset , The value of offset for the lines is not in incremental order based on line numbers in the file Is the offset the line offset or byte offset . Is there any way by which I can the get the line number of particular file on sequential order so that the lines can be sorted based on line number or offset. Currently I'm getting random values for offset for each message in a log. Can we reset the offset value to 0 so that the offset get incremented for each line of the file.