Logstash Stuck Indexing Pipeline and throwing Error - warning: already initialized constant Manticore::Client::HttpPost

Logstash was running fine and successfully reading data from the Postgres Database for indexing. However, it suddenly started throwing the following errors, causing the indexing pipeline to become stuck:

  1. logstash-8.6.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:284: warning: already initialized constant Manticore::Client::HttpPost
  2. logstash-8.6.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:536: warning: already initialized constant Manticore::Client::StringEntity

Please suggest the solution ....its urgent !!!!

Regards,
Nasim

Hello and welcome,

Please share the entire error log, just part of is not enough to troubleshoot it.

Keep in mind that there is no SLA on this forum, people will try to help when they can.

Thanks for addressing the issue!

Below is the complete log that I Logstash showing when running through window Command Line- CMD:

E:\elk_cluster\logstash-8.6.2\bin>logstash -f E:\elk_cluster\logstash-8.6.2\bin\autosuggest_index_baseline.conf
"Using bundled JDK: E:\elk_cluster\logstash-8.6.2\jdk\bin\java.exe"
Sending Logstash logs to E:/elk_cluster/logstash-8.6.2/logs which is now configured via log4j2.properties
[2023-10-27T20:29:20,658][INFO ][logstash.runner ] Log4j configuration path used is: E:\elk_cluster\logstash-8.6.2\config\log4j2.properties
[2023-10-27T20:29:20,658][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-mswin32]"}
[2023-10-27T20:29:20,658][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms8g, -Xmx8g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -agentpath:C:/PROGRA~1/DYNATR~1/oneagent/agent/bin/126717~1.202/WINDOW~2/oneagentloader.dll=loglevelcon=none,datastorage=C:\ProgramData\dynatrace\oneagent\datastorage,logdir=C:/ProgramData/dynatrace/oneagent/log]
[2023-10-27T20:29:20,725][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-10-27T20:29:22,323][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-10-27T20:29:22,592][INFO ][org.reflections.Reflections] Reflections took 172 ms to scan 1 urls, producing 127 keys and 444 values
[2023-10-27T20:29:26,169][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-10-27T20:29:26,185][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["localhost..."]} - alter this due to link upload issue
[2023-10-27T20:29:26,417][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[localhost...]}} - alter this due to link upload issue
[2023-10-27T20:29:26,801][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"localhost..."} - alter this due to link upload issue
[2023-10-27T20:29:26,828][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.6.2) {:es_version=>8}
[2023-10-27T20:29:26,829][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2023-10-27T20:29:26,851][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"intouch_autosuggest_index"}
[2023-10-27T20:29:26,854][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (data_stream => auto or unset) resolved to false
[2023-10-27T20:29:26,860][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs_compatibility => v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2023-10-27T20:29:26,886][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["E:/elk_cluster/logstash-8.6.2/bin/autosuggest_index_baseline.conf"], :thread=>"#<Thread:0x35e7ea0a@E:/elk_cluster/logstash-8.6.2/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-10-27T20:29:26,884][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2023-10-27T20:29:28,073][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.17}
[2023-10-27T20:29:28,863][INFO ][logstash.inputs.jdbc ][main] ECS compatibility is enabled but target option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the target option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-10-27T20:29:28,867][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-10-27T20:29:28,905][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2023-10-27T20:29:29,648][INFO ][logstash.inputs.jdbc ][main][14463c38da12782ddab979d1aadc13a973fc961054f67598f207c7e69230342c] (0.774781s) SELECT autosuggestid,userid,ldapalias,email, COALESCE(NULLIF(TRIM(firstname || ' ' || lastname), ''), ldapalias) AS fullname, status as userstatus FROM intouchsys2.slb_autosuggest LIMIT 999999999998
E:/elk_cluster/logstash-8.6.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:284: warning: already initialized constant Manticore::Client::HttpPost
E:/elk_cluster/logstash-8.6.2/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:284: warning: already initialized constant Manticore::Client::HttpPost
[2023-10-27T20:29:59,319][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-10-27T20:29:59,791][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-10-27T20:29:59,791][INFO ][logstash.runner ] Logstash shut down.

E:\elk_cluster\logstash-8.6.2\bin>

Regards,
Nasim

I have exactly the same warning. I could not solve it but it started when I upgraded logstash from 7.17.7 to 8.5.3 and created another "data" folder (data2) to run 2 logstash instances simultaneously.

Thanks for sharing the observation!

It is Logstash8.6.2, and there have been no recent upgrades. It was working fine for the past few months until last night. The system is a Google Cloud Platform (GCP) Virtual Machine (VM), but it's unclear whether there could be any impacts from GCP infrastructure.

The strange thing is that at the same time, the Quality Assurance (QA) environment is also experiencing the same issue, even though it had been running seamlessly alongside the Production (PROD) environment.

Regards,
Nasim

The issue still persists, but the positive aspect is that I can now ingest data successfully by refreshing the entire ELK cluster.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.