# Cant analyze byte string ( b'{... \\\\u0633 ... }' ) **URL:** \ **Category:** Logstash\ **Created:** [July 25, 2020, 4:19pm UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605 "2020-07-25T16:19:57Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![momora9809](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@momora9809](https://discuss.elastic.co/u/momora9809)\ **Post date:** [July 25, 2020, 4:19pm UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/1 "2020-07-25T16:19:57Z")
First of all, sorry for the title. I don't know what should I call that (byte string?). I have a data that is like this: ```auto b'{"id": "2", "words": ["\\u0633\\u0647\\u0627\\u0645\\u062f\\u0627\\u0631\\u06cc", "\\u0634\\u0631\\u0648\\u0639"], "content": "#\\u0648\\u0644\\u0633\\u0627\\u067e\\u0627"}' ``` The data is coming from NSQ and it is all I'm told (!). I don't know why it's like this (they said cuz of NSQ output). Logstash config: ```auto input { stdin {} } filter { bytes{ source => "message" } } output { elasticsearch { hosts => ["localhost:9200"] index => "mydata" } } ``` Analyzer in Kibana: ```auto PUT /text2 { "settings": { "analysis": { "char_filter": { "zero_width_spaces": { "type": "mapping", "mappings": ["\\u200C=>\\u0020"] } }, "filter": { "persian_stop": { "type": "stop", "stopwords": "_persian_" } }, "analyzer": { "rebuilt_persian": { "tokenizer": "standard", "char_filter": ["zero_width_spaces"], "filter": [ "asciifolding", "lowercase", "decimal_digit", "arabic_normalization", "persian_normalization", "persian_stop" ] } } } } } ``` Unfortunately, it doesn't translate "\u0633..." to Persian alphabet. But if instead, I set it like this: ```auto filter { json { source => "message" } } ``` and change the input to this: ```auto {"id": "2", "words": ["\u0633\u0647\u0627\u0645\u062f\u0627\u0631\u06cc", "\u0634\u0631\u0648\u0639"], "content": "#\u0648\u0644\u0633\u0627\u067e\u0627"} ``` it works fine. I don't know what I'm doing wrong, and again sorry I sound like a total newbie. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 26, 2020, 9:40pm UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/2 "2020-07-26T21:40:45Z")
The bytes filter does not do what you are hoping. It is used to change (for example) 1Kb into 1024. OK, so if the input line is not in the right format you can fix it. If you configure your filters as ``` mutate { gsub => [ "message", "^b'", "", "message", "'$", "", "message", "([\\])[\\]", "\1" ] } json { source => "message" } ``` you will get ``` "words" => [ [0] "سهامداری", [1] "شروع" ], "content" => "#ولساپا", ``` and Google translate confiirms that at least the words array is in Persian. The third line of the mutate, which changes \\ to \ in the string may look strange, but I can explain. You are not required to understand, or even read, the explanation to get your filter to work. In a logstash filter configuration, you cannot have \ at the end of a string, because the configuration compiler sees \" and assume it is escaping the double quote to prevent it ending the string. When replacing a backslash in a string the standard trick is to use a character group that consists of a single backslash. Thus to replace a single backslash with ! you would use ``` mutate { gsub => ["message", "[\\]", "!" ] } ``` In your case I could have use that character group and specified it should occur twice using "[\\]{2}", but then I would have to use "\" in the replacement string and we would hit the problem where you cannot have a backslash at the end of a string again. So instead I use two character groups and capture the first using () so that I can use the capture group in the replacement string (\1 refers to the first (and only) capture group). ---
**Author:** ![momora9809](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@momora9809](https://discuss.elastic.co/u/momora9809)\ **Post date:** [July 27, 2020, 3:09am UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/3 "2020-07-27T03:09:20Z")
Thank you. I'm using ```auto b'{"id": "2", "words": ["\\u0633\\u0647\\u0627\\u0645\\u062f\\u0627\\u0631\\u06cc", "\\u0634\\u0631\\u0648\\u0639"], "content": "#\\u0648\\u0644\\u0633\\u0627\\u067e\\u0627"}' ``` as input but getting this error ```auto [2020-07-27T07:37:01,215][WARN][logstash.filters.json] [main[e98b3b561145c39a1a9da545e3d90085aaead237423e9fca2815d3498d8132b5] Error parsing json {:source=>"message", :raw=>"{\"id\": \"2\", \"words\": [\"\\u0633\\u0647\\u0627\\u0645\\u062f\\u0627\\u0631\\u06cc\", \"\\u0634\\u0631\\u0648\\u0639\"], \"content\": \"#\\u0648\\u0644\\u0633\\u0627\\u067e\\u0627\"}'\r", :exception=>#}644\u0633\u0627\u067e\u0627"}' { "message" => "{\"id\": \"2\", \"words\": [\"\\u0633\\u0647\\u0627\\u0645\\u062f\\u0627\\u0631\\u06cc\", \"\\u0634\\u0631\\u0648\\u0639\"], \"content\": \"#\\u0648\\u0644\\u0633\\u0627\\u067e\\u0627\"}'\r", "@version" => "1", "host" => "DESKTOP-A", "tags" => [ [0] "_jsonparsefailure"], "@timestamp" => 2020-07-27T03:07:01.096Z } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 27, 2020, 11:21am UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/4 "2020-07-27T11:21:43Z")
> [@momora9809](#): > > `Error parsing json {:source=>"message", :raw=>"{\"id\": \"2\", \"words\": [\"\\u0633\\u0647\\u0627\\u0645\\u062f\\u0627\\u0631\\u06cc\", \"\\u0634\\u0631\\u0648\\u0639\"], \"content\": \"#\\u0648\\u0644\\u0633\\u0627\\u067e\\u0627\"}'\r",` OK, so you have a \r at the end of the input message, which means "'$" will not match, because the ' is not at the end of the string. You could try replacing the middle line of the gsub with ``` "message", "'", "", ``` ---
**Author:** ![momora9809](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@momora9809](https://discuss.elastic.co/u/momora9809)\ **Post date:** [July 28, 2020, 5:43am UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/5 "2020-07-28T05:43:12Z")
Thanks, it works! BTW, I didn't put that `\r` myself! Where did it come from?! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 28, 2020, 4:34pm UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/6 "2020-07-28T16:34:45Z")
\r is a newline. Perhaps you are processing a file with Windows line endings on a UNIX server? ---
**Author:** ![momora9809](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@momora9809](https://discuss.elastic.co/u/momora9809)\ **Post date:** [July 29, 2020, 4:27am UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/7 "2020-07-29T04:27:11Z")
I'm using Windows and both Elastic and Kibana servers are on windows too (local), but that line came from Ubuntu. Maybe that's the reason. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 26, 2020, 4:27am UTC](https://discuss.elastic.co/t/cant-analyze-byte-string-b-u0633/242605/8 "2020-08-26T04:27:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.