Issue with very long TCP messages

Elastic Stack Logstash
1

We are trying to ingest data via TCP into logstash and send to elasticsearch. Most messages are fine but sometimes our tool produces very long TCP messages and we end up with the first message getting trimmed and the remaining part of the message replaced with ... and the next message is appended to it instead of in its own message. We can't figure out why or how to fix it.

"@version" => "1",
    "@timestamp" => 2022-06-28T16:02:02.555Z,
          "host" => "10.244.0.0",
          "port" => 15655
}
{
       "message" => "<190>Jun 28 16:02:01 windows10/192.168.77.103 THOR: {\"time\":\"2022-06-28T16:02:01Z\",\"hostname\":\"windows10\",\"level\":\"Info\",\"module\":\"ProcessCheck\",\"message\":\"Process info\",\"scanid\":\"S-pKm39WX3kEo\",\"pid\":\"7700\",\"command\":\"\\\"C:\\\\Program Files\\\\WindowsApps\\\\Microsoft.WindowsStore_11910.1002.5.0_x64__8wekyb3d8bbwe\\\\WinStore.App.exe\\\" -ServerName:App.AppXc75wvwned5vhz4xyxxecvgdjhdkgsdza.mca\",\"ppid\":\"828\",\"parent\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"name\":\"WinStore.App.exe\",\"owner\":\"WINDOWS10\\\\assessor\",\"path\":\"C:\\\\Program Files\\\\WindowsApps\\\\Microsoft.WindowsStore_11910.1002.5.0_x64__8wekyb3d8bbwe\\\\WinStore.App.exe\",\"created\":\"Mon Jun 27 06:21:39 2022\",\"session\":\"Console\",\"md5\":\"6c44453cd661fc2db18e4c09c4940399\",\"connection_count\":\"0\",\"listen_ports\":\"\",\"file_1\":\"C:\\\\Program Files\\\\WindowsApps\\\\Microsoft.WindowsStore_11910.1002.5.0_x64__8wekyb3d8bbwe\\\\WinStore.App.exe\",\"exists_1\":\"yes\",\"type_1\":\"EXE\",\"size_1\":\"19456\",\"md5_1\":\"6c44453cd661fc2db18e4c09c4940399\",\"sha1_1\":\"4efe9343cefcd06ca2667da20bdd963ebb4c5a9b\",\"sha256_1\":\"fec9b389da89c183f8f1c5b96ae874e89d5284276f2cae0f1465a34d69c993fd\",\"firstbytes_1\":\"4d5a000000000000000000000000000000000000 / MZ\",\"created_1\":\"Sat Dec  7 01:54:56.861 2019\",\"owner_1\":\"BUILTIN\\\\Administrators\",\"company_1\":\"Microsoft Corporation\",\"desc_1\":\"Store\",\"legal_copyright_1\":\"Copyright ©  2015\",\"product_1\":\"Windows Store\",\"original_name_1\":\"WinStore.App.exe\",\"internal_name_1\":\"WinStore.App.exe\",\"imphash_1\":\"d41d8cd98f00b204e9800998ecf8427e\",\"file_2\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"exists_2\":\"yes\",\"type_2\":\"EXE\",\"size_2\":\"57360\",\"md5_2\":\"f586835082f632dc8d9404d83bc16316\",\"sha1_2\":\"010db07461e45b41c886192df6fd425ba8d42d82\",\"sha256_2\":\"643ec58e82e0272c97c2a59f6020970d881af19c0ad5029db9c958c13b6558c7\",\"firstbytes_2\":\"4d5a90000300000004000000ffff0000b8000000 / MZ\",\"created_2\":\"Wed Oct  6 06:51:47.630 2021\",\"owner_2\":\"NT SERVICE\\\\TrustedInstaller\",\"company_2\":\"Microsoft Corporation\",\"desc_2\":\"Host Process for Windows Services\",\"legal_copyright_2\":\"© Microsoft Corporation. All rights reserved.\",\"product_2\":\"Microsoft® Windows® Operatin...<190>Jun 28 16:02:06 windows10/192.168.77.103 THOR: {\"time\":\"2022-06-28T16:02:06Z\",\"hostname\":\"windows10\",\"level\":\"Info\",\"module\":\"ProcessCheck\",\"message\":\"Process info\",\"scanid\":\"S-pKm39WX3kEo\",\"pid\":\"8032\",\"command\":\"\\\"C:\\\\Windows\\\\SystemApps\\\\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\\\\InputApp\\\\TextInputHost.exe\\\" -ServerName:InputApp.AppX9jnwykgrccxc8by3hsrsh07r423xzvav.mca\",\"ppid\":\"828\",\"parent\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"name\":\"TextInputHost.exe\",\"owner\":\"WINDOWS10\\\\assessor\",\"path\":\"C:\\\\Windows\\\\SystemApps\\\\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\\\\InputApp\\\\TextInputHost.exe\",\"created\":\"Mon Jun 27 06:21:08 2022\",\"session\":\"Console\",\"md5\":\"4b1c22eb2fbc882f54a545d55e86491d\",\"connection_count\":\"0\",\"listen_ports\":\"\",\"file_1\":\"C:\\\\Windows\\\\SystemApps\\\\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\\\\InputApp\\\\TextInputHost.exe\",\"exists_1\":\"yes\",\"type_1\":\"EXE\",\"size_1\":\"18168\",\"md5_1\":\"4b1c22eb2fbc882f54a545d55e86491d\",\"sha1_1\":\"092a7c877829a15f539dbdacecdc0f0c4f2fbd90\",\"sha256_1\":\"43b8f0576a5d521886c489386e2030f71c981175e2d775f3513d5cb5a856ba1e\",\"firstbytes_1\":\"4d5a90000300000004000000ffff0000b8000000 / MZ\",\"created_1\":\"Wed Oct  6 06:53:05.374 2021\",\"owner_1\":\"NT SERVICE\\\\TrustedInstaller\",\"company_1\":\"Microsoft Corporation\",\"legal_copyright_1\":\"© Microsoft Corporation.  All rights reserved.\",\"product_1\":\"Microsoft® Windows® Operating System\",\"original_name_1\":\"TextInputHost.exe\",\"internal_name_1\":\"TextInputHost.exe\",\"imphash_1\":\"d41d8cd98f00b204e9800998ecf8427e\",\"file_2\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"exists_2\":\"yes\",\"type_2\":\"EXE\",\"size_2\":\"57360\",\"md5_2\":\"f586835082f632dc8d9404d83bc16316\",\"sha1_2\":\"010db07461e45b41c886192df6fd425ba8d42d82\",\"sha256_2\":\"643ec58e82e0272c97c2a59f6020970d881af19c0ad5029db9c958c13b6558c7\",\"firstbytes_2\":\"4d5a90000300000004000000ffff0000b8000000 / MZ\",\"created_2\":\"Wed Oct  6 06:51:47.630 2021\",\"owner_2\":\"NT SERVICE\\\\TrustedInstaller\",\"company_2\":\"Microsoft Corporation\",\"desc_2\":\"Host Process for Windows Services\",\"legal_copyright_2\":\"© Microsoft Corporation. All rights reserved.\",\"product_2\":\"Microsoft® W...<190>Jun 28 16:02:09 windows10/192.168.77.103 THOR: {\"time\":\"2022-06-28T16:02:09Z\",\"hostname\":\"windows10\",\"level\":\"Info\",\"module\":\"ProcessCheck\",\"message\":\"Process info\",\"scanid\":\"S-pKm39WX3kEo\",\"pid\":\"8160\",\"command\":\"C:\\\\Windows\\\\system32\\\\ApplicationFrameHost.exe -Embedding\",\"ppid\":\"828\",\"parent\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"name\":\"ApplicationFrameHost.exe\",\"owner\":\"WINDOWS10\\\\assessor\",\"path\":\"C:\\\\Windows\\\\System32\\\\ApplicationFrameHost.exe\",\"created\":\"Mon Jun 27 06:21:39 2022\",\"session\":\"Console\",\"md5\":\"d58a8a987a8dafad9dc32a548cc061e7\",\"connection_count\":\"0\",\"listen_ports\":\"\",\"file_1\":\"C:\\\\Windows\\\\system32\\\\ApplicationFrameHost.exe\",\"exists_1\":\"yes\",\"type_1\":\"EXE\",\"size_1\":\"78456\",\"md5_1\":\"d58a8a987a8dafad9dc32a548cc061e7\",\"sha1_1\":\"f79fc9e0ab066cad530b949c2153c532a5223156\",\"sha256_1\":\"cf58e424b86775e6f2354291052126a646f842fff811b730714dfbbd8ebc71a4\",\"firstbytes_1\":\"4d5a90000300000004000000ffff0000b8000000 / MZ\",\"created_1\":\"Wed Oct  6 06:51:41.051 2021\",\"owner_1\":\"NT SERVICE\\\\TrustedInstaller\",\"company_1\":\"Microsoft Corporation\",\"desc_1\":\"Application Frame Host\",\"legal_copyright_1\":\"© Microsoft Corporation. All rights reserved.\",\"product_1\":\"Microsoft® Windows® Operating System\",\"original_name_1\":\"ApplicationFrameHost.exe\",\"internal_name_1\":\"Application Frame Host\",\"imphash_1\":\"d41d8cd98f00b204e9800998ecf8427e\",\"file_2\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"exists_2\":\"yes\",\"type_2\":\"EXE\",\"size_2\":\"57360\",\"md5_2\":\"f586835082f632dc8d9404d83bc16316\",\"sha1_2\":\"010db07461e45b41c886192df6fd425ba8d42d82\",\"sha256_2\":\"643ec58e82e0272c97c2a59f6020970d881af19c0ad5029db9c958c13b6558c7\",\"firstbytes_2\":\"4d5a90000300000004000000ffff0000b8000000 / MZ\",\"created_2\":\"Wed Oct  6 06:51:47.630 2021\",\"owner_2\":\"NT SERVICE\\\\TrustedInstaller\",\"company_2\":\"Microsoft Corporation\",\"desc_2\":\"Host Process for Windows Services\",\"legal_copyright_2\":\"© Microsoft Corporation. All rights reserved.\",\"product_2\":\"Microsoft® Windows® Operating System\",\"original_name_2\":\"svchost.exe.mui\",\"internal_name_2\":\"svchost.exe\",\"imphash_2\":\"d41d8cd98f00b204e9800998ecf8427e\"}"
2

Are you certain that is happening at the receiving end? If the sender truncated the JSON at 2 KB and failed to append a newline when it was longer than that then you would get pretty much the behaviour that you are seeing.

3

I can't say for sure. Its either at the Logstash receiving end or the tool sending the TCP message. We'd have to try a different receiver to test that.

4

Looks like it was the tool, we had to add a flag to not truncate syslog messages even though it doesn't say what the default is if we don't. Thanks.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.