I have logstash running on kubernetes which gets the logs from azure-eventhub using eventhub plugin. In eventhub, the messages are logged in JSON format. While reading events from the topic in logstash and applying JSON filter or JSON codec, logstash is somehow truncating the message after every ~500 chars (JSON is valid - and works fine while posting the same using http input). The complete message is sent to elastic search when no JSON filtering is applied without any error but everything inside json appears as string.
Is there any setting the needs to be changed for filter to process the entire message and not truncate it at around 500 chars?
logstash version: docker.elastic.co/logstash/logstash-oss:7.9.0
Below is the original message getting logged in logstash using rubydebug codec in output
Original message:
"{\"EventPrimaryStampName\":\"wxxs-xxxx-mwh-0x9\",\"EventStampName\":\"wxxs-xxxx-mwh-02xf\",\"EventStampType\":\"Stamp\",\"Host\":\"XX2XXXXXXFFA\",\"category\":\"AppServiceConsoleLogs\",\"level\":\"Informational\",\"operationName\":\"Microsoft.Web/sites/log\",\"resourceId\":\"/SUBSCRIPTIONS/C4XXXXX3A-XXXD-XXXC-AF7A-XXXXXXXXX6/RESOURCEGROUPS/XXXXXXXXXXXXXX/PROVIDERS/MICROSOFT.WEB/SITES/APP1\",\"resultDescription\":\" 08:05:59 0|app | Test log message\\n\\n\",\"time\":\"2020-10-19T08:05:59.033310857Z\"}"
Truncated message:
"{\"EventPrimaryStampName\":\"wxxs-xxxx-mwh-0x9\",\"EventStampName\":\"wxxs-xxxx-mwh-02xf\",\"EventStampType\":\"Stamp\",\"Host\":\"XX2XXXXXXFFA\",\"category\":\"AppServiceConsoleLogs\",\"level\":\"Informational\",\"operationName\":\"Microsoft.Web/sites/log\",\"resourceId\":\"/SUBSCRIPTIONS/C4XXXXX3A-XXXD-XXXC-AF7A-XXXXXXXXX6/RESOURCEGROUPS/XXXXXXXXXXXXXX/PROVIDERS/MICROSOFT.WEB/SITES/APP1\",\"resultDescription\":\" 08:05:59 0|app | Test log message\\n\\n\",\"time\":\"20"[truncated 20 bytes]; line: 1, column: 333]>}
Error Message while trying to parse truncated message:
:exception=>#<LogStash::Json::ParserError: Illegal unquoted character ((CTRL-CHAR, code 27)): has to be escaped using backslash to be included in string value at [Source: (byte[])"