Filebeat is not pushing nginx logs to logstash in correct format

Elastic Stack Beats
1 
66.249.68.6, 23.59.21.50, 23.66.234.17, 104.97.76.62, 172.24.23.111, 172.24.58.27 "03/Apr/2024:03:50:44 +0530" "66.249.68.6" GET "GET /astrology/others/dream-interpretation-beginning-with-alphabet-p/articleshow/68205693.cms HTTP/1.1" 1131 200 53200 52544 - "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 10.67.7.101:80 200 0.417 0.417 0.000 0.415 - vsp1toispav2.indiatimes.com vsp1toispav2.indiatimes.com vsp1toispav2.indiatimes.com "-"

This is my nginx logs and I'm getting below error in logstash:

[main][88aef9b53ace32249a3f3afaed43c3e0ac24d6c1df8b56b3e4c7e7942709604e] Received an event that has a different character encoding than you configured. {:text=>"hY?&ln\\x89\\x96\\xECQL\\xAEh\\x90i\\x83n \\xFB\\u0014o\\xB0a\\xEE\\x87Ҕ\\xAA\\xA1v\\xDB\\\"t]u\\xF6\\u0002\\x9B\\xFF\\xD5h|\\xF6\\xFB\\xCB'\\xD3\\xDBA\\e\\x86R.h!\\x9E\\xB41s m\\x9A\\x9C\\u0014\\xD8&m\\xB2\\u0003\\xD8\\xED\\v\\u0017Y\\xF3\\xFA\\xF49\\x9C\\xCF\\xCFl\\bo\\v\\u05EE÷I\\xA8\\\"\\x9DZ\\x91BEYN,1\\xA1\\x8A]\\xED\\x87\\xC1$і\\u0013\\u0013\\xB7g\\x92\\x8C\\xF5\\xF6Ld\\u001Fiܾ\\tCs\\xE2\\xF6\\xA6٨e2\\x8Dϔ;\\x88\\xD5PEK\\u001A\\a%\\u001E\\u0015B,\\xF3]Dϓ#\\x82I;Q#h]\\xD8|\\u001A\\x87_\\xBD\\xB93\\e\\x96O\\x83\\xB1NL?\\x88z\\x81\\u0002\\xD91\\xFBN6Т\\x92\\xB4\\x98\\xED\\xBC\\xB4_\\xEA\\vLZ\\xBFh\\xA5\\xFA4\\xFD\\x82\\u001C\\u001C\\x98/0\\xC302i\\xF65U#\\x9D\\x9DtU:\\xCCN\\xF2\\xD7kWE\\xBD\\xEC\\xA6\\xEA\\xA7:\\xCE\\xCE\\xD2a\\xFE\\xC8\\xEC\\xB5\\xCF~H\\xF1\\xFA\\x83/\\u0013u\\xE7\\xE5\\x97ou\\xEF\\u000EV×_\\xDF;y\\xAD\\u00027z\\xCE\\xF8ag\\xFF\\xF4\\xD5A\\xA73\\xEA<\\xF0\\x8F\\u0006mH\\xBF\\xEDD\\xE6\\xDE\\xDD\\xFD\\x97߶\\xBF\\xFDr\\x9A\\f\\xEE\\xEC?}{\\xA4N\\xC8\\xD3\\xE8\\xFB\\xC3\\xEF\\xF8\\xF1\\xABgo\\x8E&\\x83\\xD7n\\xEA\\x9A\\xFB߭\\u0014\\xA1\\xFA\\xBC\\xDC\\u0015\\xAF6\\x802[\\u0012q\\x94\\tC؞_\\xD5j\\u0003\\x8B\\xE6\\xE4'.\\xE6\\x94Ĝ\\u000FS\\x8Cʤ\\xF8\\xD6\\xC5:\\xF2l\\xFE\\xBF\\e\\x8D\\x9D\\xCB\\xCC\\xE6\\x9F7\\u0019\\xF5\\xC4\\xE0:Ki\\xABApL &\\u001F`j\\xD5[\\xCAs\\xAEm2\\u{3CAE8}X\\u0012AThW\\xAC.\\xCF\\xE9m\\xE0E\\x84A7V+Q\\xCE\\xD7*\\x9A\\xAA\\xF8\\xCC\\xF5r\\x8D*1~\\xA0B7\\u0019\\u0005a\\x98\\xB8\\xC90\\x88\\x8A\\xB5l\\x96F\\xBD8'\\u001E\\x91+j\\u0013BLd\\f\\x9C1*=\\xAB6!+ܬW\\x9Bn\\xC5<kVF\\x8Ab\\u0002\\xED9\\x93W\\xE32o\\xD0TlK\\xE7\\xAF\\xEF\\bXu\\xA5\\xB8\\xDC\\xE6\\xA9\\xC0\\xE2\\xFF4\\u001A;/.\\xB1\\xA9\\x98\\x87`\\u0010\\xE1\\u0000\\u0013\\f\\xA4\\u0004\\xD6\\xE7\\x94\\u0014H\\x8B,\\u0004 \\x82\\u05CC\\xA3 o\\u0011\\x87\\xE0\\xDC$]\\x83C\\xBC슖.\\\\\\xD7\\f̗GqS\\u0015\\x9B\\xB4\\xFC\\x8Db\\xC5\\u0015\\xECڹ\\u0014\\u0013\\xE5k\\xAA\\xA4\\xB1", :expected_charset=>"UTF-8"}
2

Can anybody help on this ?

3

[main][88aef9b53ace32249a3f3afaed43c3e0ac24d6c1df8b56b3e4c7e7942709604e] Received an event that has a different character encoding than you configured.

:expected_charset=>"UTF-8"

  1. Make sure that FB read and send in correct character set. Set temporary output to a file
output.file:
  path: "/path/"
  filename: output.txt

Check your source to see in which the encoding format they send, might be different from UTF-8. Try with

  1. If you receiving from log file, set encoding:
    paths:
    - /path/nginx.log
    input_type: log
    encoding: "utf-8"
    If you are receiving from syslog, tcp instead of log/filestream, you should set on the source side, or again try with encoding

  2. If still doesn't help, LS input has ability to set encoding:

input {
        beats {
        port => 5044
        codec => plain { charset => "UTF-8" }
     }

Also check the documentation

4

I have already used this logstash configuration but doesn't help

5

Most likely you will have to set UTF-8 at a source.
Maybe someone will a better idea.