How to identify/display set of sequence data In the log file

Elastic Stack Logstash
1

I got one different use case from my client which I need to identify a set of log data which got recorded in sequence at any line in the log file and they want to view the identified sequence in Kibana and its related count. I am wondering whether it is possible to achieve it in Logstash filter or in Kibana query.

Below is my log file.

image

for example I have to get the count of the below sequence appeared anywhere in the log file.

  1. LS=Select
    LS=Symmetry
    LS=Select
    LS=Select
    LS=Mirror
    LS=Select

  2. LS=Select
    LS=Modify
    LS=* (which can be anything)
    LS=Select
    LS <> Select (This needs to be a command)

Please advice whether its possible to achieve.

2

I think you would need to do that in a logstash filter. The sequence of the individual log entries get lost during the ingestion of individual lines. I would suggest asking this question in the logstash forum.

3

I updated the question topic to Logstash. Thank you for showing the path.

4

At least if you use Filebeat it'll add a field to each event with the file offset of the line in question.

5

@magnusbaeck. Sorry I couldn't understand it properly can you please explain it in detail.

6

If you use Filebeat instead of Logstash for reading the files, each event will contain a field that indicates the position in the file of that line.

If this still isn't clear you'll have to explain what part you don't understand.

7

I am using filebeat but how can i achieve the sequence search using position of the file.

8

Ideally you'd just use the file offset field as the secondary sort field, but I don't think Kibana supports that. You could perhaps experiment with adding parts of the offset as the millisecond part of the timestamp.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.