Elastic Agent Filebeat - keep line order

Elastic Stack Elastic Agent
1

I'm shipping logs from Wildfly application to elasticsearch using Elastic Agent 'Custom Log File" integration.

However I've noticed that log lines are not shipped or displayed in the exact order. It probably has something to do with the way elasticsearch sorts and I'm guessing this can't be fixed, but i would like to post a question here just in case there is a solution or a workaround.

These are log lines from source log file:

2024-07-20 03:00:00,003 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 << "[read] I/O error: Read timed out"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "GET /PublicAPI/nonWorkingDays?pageSize=1000 HTTP/1.1[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "Host: node.domain.org[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "Connection: Keep-Alive[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "User-Agent: Apache-HttpClient/4.5.13 (Java/1.8.0_412)[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "[\r][\n]"

And these are log lines when looking from elasticsearch or even exporting (sorting on @timestamp)

2024-07-20 03:00:00,003 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 << "[read] I/O error: Read timed out"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "User-Agent: Apache-HttpClient/4.5.13 (Java/1.8.0_412)[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "Connection: Keep-Alive[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "Host: ebank.addiko.rs[\r][\n]"
2024-07-20 03:00:00,004 DEBUG [org.apache.http.wire] (EJB default - 1) http-outgoing-105967 >> "GET /PublicAPI/nonWorkingDays?pageSize=1000 HTTP/1.1[\r][\n]"

As you can see, when exported, line order is different from the source, and they all share the same time down to the millisecond.

Is there a way around it?

2

The two examples you shared are exactly in the same order, maybe a mistake while copying and passting?

But I think you can use the file offset to order in this case.

First sort by timestamp, then sort by the file offset.

1 Like
3

Yup, mistake when copying and pasting, thanks! I've edited the post.

Thanks for log.offset hint! I remember there used to be someething like this with filebeat but somehow when i searched for the fields I've missed it.

When I'm using it in conjunction with @timestamp it seems to work just fine. Thanks a lot! :slight_smile: