Logs not in sequence as in console output

Elastic Stack Logstash
1

I am sending logs from Jenkins to ELK. Because of the jade timestamp and not having enough precision to nano or milli seconds , I have introduced a sequence number against every line of log that enters logstash . Even after introducing the sequence number , the problem persists. If the logs are not in sequence in Kibana , its misleading to the users.

Below is the sample of console logs in jenkins :

Capture2
Capture2.PNG749×384 43.1 KB

Below is a sample of how logs are in my kibana : The number on extreme right is the sequence number.

image
image.png1337×716 24.3 KB

As can be seen , the log line "Finished :Success " is at the bottom in actual jenkins logs . but in my kibana UI it appears in the middle.

Will any change in the pipeline work ?

Thanks in advance.

2

How are you assigning the sequence number? If done in a filter it is possible the logs are not processed in order. I believe Filebeat provides an offset value that you can use to order data within a file, so I would recommend using that over the Logstash file input plugin.

3

Offset is the number of bytes read. If I sort by offset , the sequence will not be maintained. I tried doing that as I can see a filed by name "offset" in kibana.

I am using input filter , and ruby code to introduce the sequence number to every line of log. As can be seen below , the sequence starts from 100 and increases by one to every line of log , it encounters.

image

4

The offset will give you the order although not sequential values. I am not sure if the ordering of events is guaranteed once you get to the filter stage so you may need to implement a custom codec.

5

With the java execution engine (the default in v7) both filters and outputs can re-order events, even with '--pipeline.workers 1'.

6

Thanks for responding back . Glad to know about the issue with V7. Even though I am not using V7 , if the order is not maintained , its misleading. There must be a fix for this.

7

Right now I have reduced 'pipeline.batch.size' to 1 . After this the order is maintained in Kibana , how ever only time will tell how effective a solution this is . I have not changed the default pipeline workers , which is 1 by default. I am not able to figure out how to use '-w 1'. my ELK version is 6.3.2

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.