hi, we are running into parsing errors when sending data through logstash pipelines. After digging a little, found that the source system has been sending a lot of unicode special characters - something like below: \u0006\u0000\u0000\u0000\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0000\u0000\u0000…

I could really use some guidance with the above issue. Any suggestions please?

Have you tried mutate gsub filter: filter { mutate { gsub => [ "fieldname", "[\u0000\u0001]", "", ] } } ?

Thanks Tomo. Yes, I tried using gsub filter quite often in the past. But in this particular case, these unicode strings could vary with each different character and so I was looking for something generic to use. The fieldnames could be \u0000 or any other character/number after "\u". It will be high…

[image] vee: unicode special characters Are there so many special characters? How to distinguish from useful characters.

That's where I was looking to get some help :slight_smile: I was hoping there is some readily available filter to tackle those unicode characters.

How about: filter { mutate { gsub => [ "fieldname", "[\u0000-\u001F]", "", ] } } ? There was no ignore option on plain codec plugin. So it seems you have to specify the characters by regex. I suppose it is not so many special characters generated from…

No luck yet. Still errors in parsing the message: at [Source: (byte[])"{"id":"79f9856b-ba63-41fd-9665-b8484086cb5c","message":"Claimed ....

Can you share the full error you are receiving? Is that unicode string part of your message or it is your entire message this way? Is every message like this or just some random message will give you this error? Can you share an example of a message that is giving you the error? What is your logs…

Here is the sample test message (removed sensitive info): {"app":"test-api[v1]","hostname":"test-api-blue04-dc1-zn1","port":10610,"@timestamp":"2022-02-10T10:56:29.205-06:00","logLevel":"INFO","threadName":"Default Executor-thread-2453","loggerName":"com.rest.util.RestLoggingHelper","message":"rest…

Logstash parshing unicode characters

Elastic Stack Logstash

Tomo_M (Tomohiro Mitani) February 15, 2022, 7:48pm 13

try gsub before json filter..

Topic		Replies	Views
Remove Special character from message before sending to json filter Logstash	5	378	August 2, 2022
Problem witch characters "ãˆ°ãˆ±â´±ãˆã„¶ã„¶ã„°" in field Message Logstash	3	319	January 13, 2022
Logstash throwing error .Unexpected character (‘<’ (code 60)) Logstash	12	3565	September 18, 2019
Json { source => "message" } not parsing all of it Logstash	21	1959	March 22, 2022
How to handle special characters (hex encoded) in logstash mutate or grok Logstash	7	434	March 14, 2024

Logstash parshing unicode characters

Related topics