Regarding Filebeat Offset Value

Elastic Stack Beats
#1

Regarding filebeat offset , The value of offset for the lines in the same log file is not shown in sequential order based on order of line numbers in the file.

I've some queries on the filebeat offset .

Is the offset the line offset or byte offset .
Is there any way by which I can the get the line number of particular file on sequential order so that the lines can be sorted based on line number or offset.
Can we reset the offset value to 0 so that the offset get incremented after reading each line of the file.

#2

Offset is the number of bytes read. For more details on why there is no line number see here: https://github.com/elastic/beats/issues/1037

As the offset is used for internal logic, there is no overwriting possible.

1 Like
#3

Thanks for the information.

Could you please let me know if there is any field in filebeat or logstash based on which we can sort the lines in the a particular file.

Based on the line selected we should be able to filter number of lines above and below that line in the file.

#4

Is there any field in filebeat which sequencially get incremented on reading of each line in a log file and reset it when the next file is read.

#5

I think what you are looking for is offset. You mentioned in the beginning that it is not necessarly the same order, but that shouldn't be. If that is the case, can you share some more details?

#6

in this case, can we filter by ip and file path.
then order by timestamp and offset ?

#7

@iglxxx Probably i need some more details from you here. Can you open a new topic?

#8

en. there is a case in our company. we use filebeat collect log file and send to elasticserch.
we search something from elasticsearch, if we select one result, we want to see 10 lines above it and 10 lines before it according the order in the original log file.
in the first time, we sort by timestamp ,but it does not work well, beacuse timestamp my be the same value.
in the second time, we sort by offset only, but it has the same value.
at the end, we sort by both timestamp and offset, it seems work well.
but i find a pull in kibana can solve it.

#9

So the problem is solved for you? If not, please open a new topic as this one is already a year old. Closing the topic.

#10