Issue with Extracting "trace_id" from Logs in Elasticsearch Using Filebeat Pipeline

Elastic Stack Kibana
1

I am currently pushing logs to the Elasticsearch from Filebeat. I’m trying to extract the trace_id from the message field and store the value in a separate field.

The message field has the value: 2024-09-23 10:00:915 [http-nio-9100-exec-1] [INFO ] com.ZZZ.YYY.service.UserService [XXXX-80c4-437c-b7ae-XXXXX,YYYYYY-4e90-474e-ba30-YYYYY] [ZZZ-services,post/MMM/process] : Getting user by uuid <getUserByUUID>YYYYYY-4e90-474e-ba30-YYYYY

I created an ingest pipeline with a Grok pattern to capture the trace_id, but it doesn’t appear in Kibana. I have added the pipeline name in the filebeat configuration as well.

Grok pattern: %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{NUMBER:milliseconds} \[%{DATA:thread}\] \[%{LOGLEVEL:loglevel}\s\] %{JAVACLASS:class} \[%{UUID:trace_id},%{UUID:other_id}\] \[%{DATA:service}\] : %{GREEDYDATA:message}

When I tested the ingest pipeline using a document, it was able to extract is successfully.

My question is why is the trace_id field not appearing in Kibana? Is there any additional configurations to be done? Thanks in advance.

2

The grok pattern cannot work because:

  • missing \s+ after loglevel
  • UUID must be in this range [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}, XXXXX is not allowed. Use DATA or define your own pattern

%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{NUMBER:milliseconds} \[%{DATA:thread}\] \[%{LOGLEVEL:loglevel}\s*\] %{JAVACLASS:class} \[%{DATA:trace_id},%{DATA:other_id}\] \[%{DATA:service}\] : %{GREEDYDATA:message}

3

Thanks @Rios . I have updated the Grok pattern and even verified it with a sample document. But still I am not able to see the field created. Is there anything else to be done? New index has to be created for the changes to reflect?

4

Not sure which version ELK do you have, for lower than 8.x you have to manually refresh index patterns which is now named: the data view.7

If is a test env, drop index and recreate data view.

Another possibility is that your messages have optioned fields, check in ruby debug.

5

Could you try to have a look on Data view? Or just try to create a new data view, It should visible.

6

I am using ES version 8.5.1. I removed the data view and created a new one. Still not able to find the trace_id field.