Dissect Processor not working correctly in elastic search

Hi Team,

I am trying to parse logs using dissect processor in elastic search . However I dont get the appropriate output. When I run the log through logstash using dissect filter, it works fine.

`

PUT _ingest/pipeline/my_pipeline_id_saket
{
"description" : "describe pipeline",
"processors" : [
{
"dissect": {
"field": "message",
"pattern": "%{log_TraceId}, %{log_SpanId}, %{log_UserId}, [%{ip},"
}
}
]

}

POST test/_doc/3?pipeline=my_pipeline_id_saket
{
"message" : "b5721cae-573f-4da5-90dd-2f1c7783b21d, b5721cae-573f-4da5-90dd-2f1c7783b21d, , [0:0:0:0:0:0:0:1,"

}

GET test/_doc/3

`

This gives the output :-

"log_TraceId" : "b5721cae-573f-4da5-90dd-2f1c7783b21d", "log_UserId" : "", "ip" : "[0:0:0:0:0:0:0:1", "log_SpanId" : "b5721cae-573f-4da5-90dd-2f1c7783b21d"
However the output in logstash (correct & expected):-
{ "ip": "0:0:0:0:0:0:0:1", "log_SpanId": "b5721cae-573f-4da5-90dd-2f1c7783b21d", "log_TraceId": "b5721cae-573f-4da5-90dd-2f1c7783b21d", "log_UserId": "" }

The Ip address does not get parsed correctly and has a '[' added to it which is a delimiter.

Thanks
Saket

If I recall correctly I think the dissect pattern need to end with a match clause, so I would recommend adding and empty one at the end and see if that makes a difference.

Thanks for your reply. Sorry I could not understand " adding and empty one at the end and see if that makes a difference.". Could you please give an example ?

Was on my mobile so could not provide an example. Meant something like this:

"pattern": "%{log_TraceId}, %{log_SpanId}, %{log_UserId}, [%{ip},%{}"

THanks for your reply. I just did try that unfortunately I still get a '[' in my ip

If the same config in Logstash produces a different result it sounds like a bug.

Yes it does..!! Thanks for your reply.

Hi,

Humble Request, if anyone from Elastic Team can look into this ?

Thanks
Saket

If it's a bug, could you open an issue in elasticsearch GitHub repository?

You should link to this thread from the issue.