After happily deploying ElasticSearch Upgrade to 5.6.1 with only minor issues, got in this morning to find that all TCP input pipelines for logging messages that do not adhere to the requirements by the syslog input had stopped working since the upgrade. After some digging I came across this discussion: Logstash - wrong codec in tcp input plugin?
Which pointed me to the issue I was experiencing. Basically the debugging output of LOGSTASH would show me that my TCP inputs were using the LINE codec even if I configurated explicitly PLAIN since my log messages do not feature newline characters. Here a few of those logging lines:
[2017-09-25T09:45:12,266][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@id = "line_07232e41-f78d-4aed-a386-06a8134ccf6b"
[2017-09-25T09:45:12,266][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@enable_metric = true
[2017-09-25T09:45:12,267][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@charset = "UTF-8"
[2017-09-25T09:45:12,267][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@delimiter = "\n"
After reading the above quoted discussion + explanation by @pierhugues I understand the code changes introduced here: LINK to #fix_streaming_codecs fix in inputs/base.rb to some extend, leaving the particulars to users that require this change.
BUT: this is a breaking change for anyone that is processing logging messages that cannot be '\n' terminated or terminated in any shape or form and have to be received via TCP not UDP. In my particular case I use haproxy as a load balancer in front of two ingest nodes and haproxy only does TCP...
Anyway, I would like to motion if it would be possible to have a simple PLAIN_FORCED codec that users can define in the input configuration if they are certain that they want to have plain plain messages without terminating characters and no surprises?! E.g. that would not be replaced rather noiselessly in the background. I could obviously just create my own plugin, but am sure that others might be interested as well + active developers already have all tools setup and the required coding is minimal etc.
Cheers,
Gabriel