We have a simple pipeline that has been running fine for weeks without any errors, and then suddenly stopped outputting events while repeatedly printing this error:
{"level":"ERROR","loggerName":"logstash.agent","timeMillis":1631284844013,"thread":"Converge PipelineAction::Reload<metric-tracking>","logEvent":{"message":"Failed to execute action","id":{"metaClass":{"metaClass":{"metaClass":{"id":"metric-tracking","action_type":{"metaClass":{"metaClass":{"action_type":"LogStash::ConvergeResult::FailedAction","message":"Couldn't find any output plugin named 'http'. Are you sure this is correct? Trying to load the http output plugin resulted in this error: Block validation fails for plugin named http of type output,","backtrace":["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in
lookup_pipeline_plugin'","/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'","org/logstash/plugins/PluginFactoryExt.java:200:in
plugin'","org/logstash/plugins/PluginFactoryExt.java:137:in buildOutput'","org/logstash/execution/JavaBasePipelineExt.java:50:in
initialize'","/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:37:in execute'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in
block in converge_state'"]}}}}}}}}}`
At this point, the pipeline totally stopped working. After restart, it worked again.
The pipeline is very simple:
input {
file {
id => "tracking_file_logs_buffer"
path => "/mnt/logstash/data/logs/logstash-buffer-*.log"
sincedb_path => "/mnt/logstash/data/plugins/inputs/file/sincedb_tracking"
codec => json
}
}
output {
http {
id => "tracking-output"
http_method => "post"
url => "http://some.url:8084/v1/activity/ingest"
format => json_batch
pool_max => 1
}
}
There was nothing obvious that triggered that error. Memory and cpu looked fine.
Any idea what could be the issue? How can we prevent this error from happening again?
Logstash version: 6.8.17
Linux version: 5.10.50-44.132.amzn2.x86_64