How to identify/display set of sequence data In the log file

I got one different use case from my client which I need to identify a set of log data which got recorded in sequence at any line in the log file and they want to view the identified sequence in Kibana and its related count. I am wondering whether it is possible to achieve it in Logstash filter or in Kibana query.

Below is my log file.

image

for example I have to get the count of the below sequence appeared anywhere in the log file.

  1. LS=Select
    LS=Symmetry
    LS=Select
    LS=Select
    LS=Mirror
    LS=Select

  2. LS=Select
    LS=Modify
    LS=* (which can be anything)
    LS=Select
    LS <> Select (This needs to be a command)

Please advice whether its possible to achieve.

I think you would need to do that in a logstash filter. The sequence of the individual log entries get lost during the ingestion of individual lines. I would suggest asking this question in the logstash forum.

I updated the question topic to Logstash. Thank you for showing the path.

At least if you use Filebeat it'll add a field to each event with the file offset of the line in question.

@magnusbaeck. Sorry I couldn't understand it properly can you please explain it in detail.

If you use Filebeat instead of Logstash for reading the files, each event will contain a field that indicates the position in the file of that line.

If this still isn't clear you'll have to explain what part you don't understand.

I am using filebeat but how can i achieve the sequence search using position of the file.

Ideally you'd just use the file offset field as the secondary sort field, but I don't think Kibana supports that. You could perhaps experiment with adding parts of the offset as the millisecond part of the timestamp.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.