I got one different use case from my client which I need to identify a set of log data which got recorded in sequence at any line in the log file and they want to view the identified sequence in Kibana and its related count. I am wondering whether it is possible to achieve it in Logstash filter or in Kibana query.
Below is my log file.
for example I have to get the count of the below sequence appeared anywhere in the log file.
-
LS=Select
LS=Symmetry
LS=Select
LS=Select
LS=Mirror
LS=Select
-
LS=Select
LS=Modify
LS=* (which can be anything)
LS=Select
LS <> Select (This needs to be a command)
Please advice whether its possible to achieve.
I think you would need to do that in a logstash filter. The sequence of the individual log entries get lost during the ingestion of individual lines. I would suggest asking this question in the logstash forum.
I updated the question topic to Logstash. Thank you for showing the path.
At least if you use Filebeat it'll add a field to each event with the file offset of the line in question.
@magnusbaeck. Sorry I couldn't understand it properly can you please explain it in detail.
If you use Filebeat instead of Logstash for reading the files, each event will contain a field that indicates the position in the file of that line.
If this still isn't clear you'll have to explain what part you don't understand.
I am using filebeat but how can i achieve the sequence search using position of the file.
Ideally you'd just use the file offset field as the secondary sort field, but I don't think Kibana supports that. You could perhaps experiment with adding parts of the offset as the millisecond part of the timestamp.