# Management of illegal characters **URL:** \ **Category:** Logstash\ **Created:** [April 15, 2025, 4:36pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159 "2025-04-15T16:36:53Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Francesco\_Esposito](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francesco_esposito/32/141865_2.png) [@Francesco\_Esposito](https://discuss.elastic.co/u/Francesco_Esposito)\ **Post date:** [April 15, 2025, 4:36pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/1 "2025-04-15T16:36:53Z")
Hello Everyone, I am in the process of converting a "in-house" Delphi application that receives an XML string over udp port and send it to an elasticsearch instance, doing a conversion from XML to JSON and sending it thru \_bulk api. Now, I am trying to receive an XML string over UPD Input Plugin, and happens that this XML string contains illegal characters, like #0 (Nul string termination). I am using this configuration: ```auto input { udp { port => 517 } } filter { xml { force_array => false source => "message" target => "myxml" } } output { file { path => "/log_streaming/my_app/records/log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log" codec => line { format => "%{myxml}" } } } ``` And when everything is good I can receive data in this format: ```auto { "APPVERSION": "1.0.1.11", "EVENTDATETIME": "04/14/2025 18:38:20:203", "EVENTNAME": "TestEvent\n2\n04/14/2025 18:38:20:38", "APPLICATION": "TESTUDPLOGGER", "HOST": "FRANCESCOE-RMT", "EVENTINFO": "04/14/2025 18:38:20:38", "LINENO": "1", "INSTANCEID": "BD2051FB-525E-49CD-BEDB-3DEF967ADCFB", "SEVERITY": "0", "THREADID": "13852", "EVENTSEQNO": "1" } ``` But if a #0 is received, I have this error: ```auto Illegal character "\u0000" in raw string "04/15/2025 11:55:55:55\u0000ben\u0000frank\u0000sue" ``` Writing to the disc what I receive (removing "filter" part) I receive: ```auto {"@timestamp":"2025-04-14T17:52:37.590740500Z","event":{"original":"FRANCESCOE-RMT65C3FEC1-B288-437F-B0C3-8CA3EB1956ECTESTUDPLOGGER70801.0.1.111104/14/2025 13:52:37:5870TestEvent\r\n1\r\n04/14/2025 13:52:37:5204/14/2025 13:52:37:52\u0000ben\u0000frank\u0000sue"},"host":{"ip":"127.0.0.1"},"@version":"1","message":"FRANCESCOE-RMT65C3FEC1-B288-437F-B0C3-8CA3EB1956ECTESTUDPLOGGER70801.0.1.111104/14/2025 13:52:37:5870TestEvent\r\n1\r\n04/14/2025 13:52:37:5204/14/2025 13:52:37:52\u0000ben\u0000frank\u0000sue"} ``` As you can see, it was converted in "\u0000". I need to convert #0, #13#10, #13 and #10 to a one character space. How can I do that? ---
**Author:** ![Francesco\_Esposito](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francesco_esposito/32/141865_2.png) [@Francesco\_Esposito](https://discuss.elastic.co/u/Francesco_Esposito)\ **Post date:** [April 15, 2025, 4:54pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/2 "2025-04-15T16:54:16Z")
Sorry everyone, it was simply: ```auto input { udp { port => 517 } } filter { mutate { gsub => ["message", "\u0000", "[0x00]" ] } mutate { gsub => ["message", "\r\n", "[0x01]" ] } mutate { gsub => ["message", "\r", "[0x02]" ] } mutate { gsub => ["message", "\n", "[0x03]" ] } xml { force_array => false source => "message" target => "myxml" } } output { file { path => "/log_streaming/my_app/records/log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log" codec => line { format => "%{myxml}" } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 15, 2025, 5:26pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/3 "2025-04-15T17:26:41Z")
> [@Francesco\_Esposito](#): > > I need to convert #0, #13#10, #13 and #10 to a one character space. How can I do that? ``` mutate { gsub => ["message" , "[\r\n^@]", " " ] } ``` That ^@ is a literal NUL character. In vim I can type it using Ctrl/v Ctrl/Shift/2 I then get ``` "EVENTNAME" => "TestEvent1 04/14/2025 14:11:24:11", "EVENTINFO" => "04/14/2025 14:11:24:11 ben frank sue" ``` using the example from your other thread. ---
**Author:** ![Francesco\_Esposito](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francesco_esposito/32/141865_2.png) [@Francesco\_Esposito](https://discuss.elastic.co/u/Francesco_Esposito)\ **Post date:** [April 15, 2025, 7:08pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/4 "2025-04-15T19:08:10Z")
Okay, that's good @Badger . I have used my formatting to remain consistent with what I had before in my delphi application, but your solution works. ```auto { "APPVERSION": "1.0.1.12", "EVENTDATETIME": "04/15/2025 15:00:41:955", "EVENTNAME": "TestEvent[0x01]9[0x01]04/15/2025 15:00:41:00", "APPLICATION": "TESTUDPLOGGER", "EVENTINFO": "04/15/2025 15:00:41:00[0x00]ben[0x01]frank[0x02]sue[0x03]john", "HOST": "FRANCESCOE-RMT", "LINENO": "1", "INSTANCEID": "CA61547F-3154-48A9-A50D-E650D9243F8E", "SEVERITY": "0", "THREADID": "16912", "EVENTSEQNO": "1" } ``` This is my file now. If I want to send it to elasticsearch, shouldn't be enough to activate elasticsearch plugin like this? ```auto input { udp { port => 517 } } filter { mutate { gsub => ["message", "\u0000", "[0x00]" ] } mutate { gsub => ["message", "\r\n", "[0x01]" ] } mutate { gsub => ["message", "\r", "[0x02]" ] } mutate { gsub => ["message", "\n", "[0x03]" ] } xml { force_array => false source => "message" target => "myxml" } } output { file { path => "/log_streaming/my_app/records/log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log" codec => line { format => "%{myxml}" } } elasticsearch { hosts => "https://myhost:9200" user => "myuser" password => "mypassword" ssl_certificate_verification => "false" index => "delphi-processlog_write" codec => line { format => "%{myxml}" } } } ``` Because I see my data inserted in fields called "myxml.my fields". ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 15, 2025, 7:46pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/5 "2025-04-15T19:46:15Z")
> [@Francesco\_Esposito](#): > > `codec => line { format => "%{myxml}" }` Every output allows you to specify the codec option because it is [implemented](https://github.com/elastic/logstash/blob/49cf7acad0274f605d0a11b9e9ddcbfb4c764100/logstash-core/lib/logstash/outputs/base.rb#L35) by the base codec class that they extend. But that doesn't mean they use the codec to send the event to the destination. The elasticsearch output ignores the codec option and [formats](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/dac08b2ea9623965166a15c98ba9fed8d4ea55d2/lib/logstash/outputs/elasticsearch/http_client.rb#L132) the event as a JSON string, because that's what the \_bulk API requires. If you use store\_xml you have to specify a target, if you want to move the fields back up to the top-level of the event then see [this](https://discuss.elastic.co/t/how-to-dynamically-move-nested-key-value-to-root-level/180006/2) thread. ---
**Author:** ![Francesco\_Esposito](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francesco_esposito/32/141865_2.png) [@Francesco\_Esposito](https://discuss.elastic.co/u/Francesco_Esposito)\ **Post date:** [April 15, 2025, 8:35pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/6 "2025-04-15T20:35:30Z")
It works perfectly, thanks! Last question @Badger , I receive a date in this format: ```auto 04/15/2025 16:28:09:361 ``` That gives me this error: ```auto Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"delphi-processlog_write", :routing=>nil}, {"eventseqno"=>"1", "exeversion"=>"1.0.1.12", "severity"=>"0", "@timestamp"=>2025-04-15T20:28:09.363306700Z, "apphost"=>"FRANCESCOE-RMT", "exename"=>"TESTUDPLOGGER", "eventname"=>"TestEvent[0x01]5[0x01]04/15/2025 16:28:09:28", "logeventdate"=>"04/15/2025 16:28:09:361", "eventinfo"=>"04/15/2025 16:28:09:28[0x00]ben[0x01]frank[0x02]sue[0x03]john", "instanceid"=>"0518D8D7-28CD-4AF3-99DC-8D4CE30CCC6E", "threadid"=>"23476"}], :response=>{"index"=>{"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [logeventdate] of type [date] in document with id 'GkcjO5YBP4xpwjYARoME'. Preview of field's value: '04/15/2025 16:28:09:361'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [04/15/2025 16:28:09:361] with format [yyyy/MM/dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}} ``` How would you make elasticsearch accept this format or how to convert it? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 15, 2025, 9:52pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/7 "2025-04-15T21:52:08Z")
Your elasticsearch index is configured to expect a logeventdate in one of the formats ``` yyyy/MM/dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time||epoch_millis ``` Your field actually has the format `yyyy/MM/dd HH:mm:ss:SSS`. I don't know if you can add an additional date format to an existing index (obviously you can update the index template so that future indexes will accept that format). Otherwise change the colon before the milliseconds to a stop using ``` mutate { gsub => ["EVENTDATETIME", "(\d{2}):(\d{3})$", "\1.\2"] } ``` ---
**Author:** ![Francesco\_Esposito](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francesco_esposito/32/141865_2.png) [@Francesco\_Esposito](https://discuss.elastic.co/u/Francesco_Esposito)\ **Post date:** [April 15, 2025, 10:08pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/8 "2025-04-15T22:08:50Z")
Unfortunately, my format looks like being ```auto MM/dd/yyyy HH:mm:ss:SSS ``` So that's not enough... ☹ ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 15, 2025, 11:46pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/9 "2025-04-15T23:46:18Z")
You could parse it using a [date](https://www.elastic.co/docs/reference/logstash/plugins/plugins-filters-date) filter. I believe that will get sent to elasticsearch in an acceptable format. If not, you can reformat it using ruby. See [this](https://discuss.elastic.co/t/date-field-being-converted-to-timestamps/180044/4) thread. You could also reformat it using a more complex gsub, but that doesn't feel right to me. ``` mutate { gsub => ["someField", "(\d{2})/(\d{2})/(\d{4}) (\d{2}:\d{2}:\d{2}):(\d{3})", "\3/\1/\2 \4.\5"] } ``` Just looking at that makes my eyes bleed! ---
**Author:** ![Francesco\_Esposito](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francesco_esposito/32/141865_2.png) [@Francesco\_Esposito](https://discuss.elastic.co/u/Francesco_Esposito)\ **Post date:** [April 30, 2025, 2:56pm UTC](https://discuss.elastic.co/t/management-of-illegal-characters/377159/10 "2025-04-30T14:56:40Z")
At the end I have used the date filter plugin: ```auto date { match => ["logeventdate", "MM/dd/yyyy HH:mm:ss:SSS"] timezone => "Etc/GMT+4" target => "logeventdate" } ``` It worked just fine.