Input STDIN 4096 character input limit?

dmg0122 · May 21, 2016, 6:40pm

File wc:

#] wc file
   1   22 4097 file

Processing some json from the STDIN input I get the following:

:exception=>#<LogStash::Json::ParserError: Unexpected end-of-input in VALUE_STRING
at [Source: [B@1d6f5292; line: 1, column: 8191]>, :level=>:warn}

After cutting off sections of the data piece by piece I was able to get it to successfully output to the stdout rubydebug codec when the character count was at or below 4096.

Is there a known issue in regards to large sets of json data being pass through the STDIN input?

Tried on logstash latest stable and 5.0.2 in docker and also outside of docker.

raivis.krumins · February 24, 2017, 12:47pm

Hi!

It looks like this limit is hard-coded in plugin source (size in bytes):

github.com

logstash-plugins/logstash-input-stdin/blob/v3.2.2/lib/logstash/inputs/stdin.rb#L17




# Read events from standard input.
#
# By default, each event is assumed to be one line. If you
# want to join lines, you'll want to use the multiline codec.
class LogStash::Inputs::Stdin < LogStash::Inputs::Base
config_name "stdin"


default :codec, "line"


READ_SIZE = 16384


def register
  begin
    @stdin = StdinChannel::Reader.new
    self.class.module_exec { alias_method :stdin_read, :channel_read }
    self.class.module_exec { alias_method :stop, :channel_stop}
  rescue => e
    @logger.debug("fallback to reading from regular $stdin", :exception => e)
    self.class.module_exec { alias_method :stdin_read, :default_read }
    self.class.module_exec { alias_method :stop, :default_stop }