Long message is truncated by logstash

Elastic Stack Logstash
1

</ Using bundled JDK: /opt/logstash-8.17.2/jdk Sending Logstash logs to /opt/logstash-8.17.2/logs which is now configured via log4j2.properties [2025-02-18T22:38:22,680][WARN ][logstash.runner ] NOTICE: Running Logstash as a superuser is strongly discouraged as it poses a security risk. Set 'allow_superuser' to false for better security. [2025-02-18T22:38:22,686][INFO ][logstash.runner ] Log4j configuration path used is: /opt/logstash-8.17.2/config/log4j2.properties [2025-02-18T22:38:22,687][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead. [2025-02-18T22:38:22,688][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.17.2", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.6+7-LTS on 21.0.6+7-LTS +indy +jit [x86_64-linux]"} [2025-02-18T22:38:22,690][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms8g, -Xmx8g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=30000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11] [2025-02-18T22:38:22,720][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override logstash.jackson.stream-read-constraints.max-string-lengthconfigured to200000000[2025-02-18T22:38:22,720][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value overridelogstash.jackson.stream-read-constraints.max-number-lengthconfigured to30000[2025-02-18T22:38:22,826][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2025-02-18T22:38:23,166][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9603, :ssl_enabled=>false} [2025-02-18T22:38:23,290][INFO ][org.reflections.Reflections] Reflections took 81 ms to scan 1 urls, producing 152 keys and 530 values [2025-02-18T22:38:23,427][INFO ][logstash.javapipeline ] Pipelinemainis configured withpipeline.ecs_compatibility: v8setting. All plugins in this pipeline will default toecs_compatibility => v8 unless explicitly configured otherwise. [2025-02-18T22:38:23,449][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>32, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>4000, "pipeline.sources"=>["/opt/logstash-8.17.2/config/test2.conf"], :thread=>"#<Thread:0x64d58fc6 /opt/logstash-8.17.2/logstash-core/lib/logstash/java_pipeline.rb:138 run>"} [2025-02-18T22:38:24,022][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.57} [2025-02-18T22:38:24,045][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} The stdin plugin is now waiting for input: [2025-02-18T22:38:24,058][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} >

this is my test file:

input {
stdin {
}
}

output {
stdout {
codec => rubydebug
}
}`

input message is : {"testesteste":{"domain":"testestestetestestestetestestestetestestestetestestestetestestestetestesteste","request_url":"/api/v1/prtestestestete","@version":"1","http_referer":"-","via":"-","protocol":"http","status":"200","client_ip":"10.26.2.2.","upstream_addr":"1.1.1.1","size":"0","xff":"-","@timestamp":"2025-02-18T22:39:05+08:00","remote_user":"-","request_method":"POST","slb_req_id":"","host":"abidahidnifasfdasfda","user_agent":"Prometheus/2.28.1testestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestesteste2.28.1testestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestesteaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa2.28.1testestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestesteste2.28.1testestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestestetestestesteaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","cookie_id":"-","upstream_status":"200","request_time":"0.014","idc_tag":"dsda","cluster":"dfaasfdasfdsaf","upstream_response_time":"0.013"}}

1739889917088
17398899170882517×944 83.9 KB

you can find the message was truncated by filter plugins or others

How do I fix this? Are there any restrictions that can be adjusted?

2

Try to set params in jvm.options:

-Dlogstash.jackson.stream-read-constraints.max-nesting-depth=1000
-Dlogstash.jackson.stream-read-constraints.max-string-length=200000000
3

Your message is 7447 bytes long. If I put that in a file and read it using a stdin input then

input { stdin { } }
filter { ruby { code => 'puts event.get("message").length' } }
output { stdout { codec => rubydebug } }

tells me message is 7447 bytes. However, if I run logstash and let it read from the tty, if I paste the 7447 byte message into the terminal then [message] is 4095 bytes. It's not breaking the line into two, it's just truncating it.

According to this SE post, you can change this using stty -icanon icrnl, and it works for me (make sure you understand what it is turning off). This is all nothing to do with logstash, it's a limit on cut and paste in a tty.

icanon keeps getting re-enabled for me, so the 4095 limit keeps returning. Not sure why.

1 Like
4

Badger, can be this issue?

5

I don't think those messages are related to this problem.

This issue on github links to other issues and PRs related to why these get logged. I didn't dig through the history to see when they started being logged. I think it was recently (a few months at most), but I could be wrong.

1 Like
6

thank you so mush! I understand what you mean. It's indeed the issue of tty restrictions.