Suddenly pipeline got stuck with error couldn't find any output plugin named http

[michael.l]

We have a simple pipeline that has been running fine for weeks without any errors, and then suddenly stopped outputting events while repeatedly printing this error:

{"level":"ERROR","loggerName":"logstash.agent","timeMillis":1631284844013,"thread":"Converge PipelineAction::Reload<metric-tracking>","logEvent":{"message":"Failed to execute action","id":{"metaClass":{"metaClass":{"metaClass":{"id":"metric-tracking","action_type":{"metaClass":{"metaClass":{"action_type":"LogStash::ConvergeResult::FailedAction","message":"Couldn't find any output plugin named 'http'. Are you sure this is correct? Trying to load the http output plugin resulted in this error: Block validation fails for plugin named http of type output,","backtrace":["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in lookup_pipeline_plugin'","/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'","org/logstash/plugins/PluginFactoryExt.java:200:in plugin'","org/logstash/plugins/PluginFactoryExt.java:137:in buildOutput'","org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'","/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:37:in execute'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in block in converge_state'"]}}}}}}}}}`

At this point, the pipeline totally stopped working. After restart, it worked again.

The pipeline is very simple:

input {
  file {
    id => "tracking_file_logs_buffer"
    path => "/mnt/logstash/data/logs/logstash-buffer-*.log"
    sincedb_path => "/mnt/logstash/data/plugins/inputs/file/sincedb_tracking"
    codec => json
  }
}

output {
      http {
      id => "tracking-output"
      http_method => "post"
      url => "http://some.url:8084/v1/activity/ingest"
      format => json_batch
      pool_max => 1
    }
  }

There was nothing obvious that triggered that error. Memory and cpu looked fine.

Any idea what could be the issue? How can we prevent this error from happening again?

Logstash version: 6.8.17
Linux version: 5.10.50-44.132.amzn2.x86_64

[grumo35]

Hi,

@michael.l

Logstash version: 6.8.17

Why not try to update first ? your logstash version is very old this could be a lot of things.

[michael.l]

@grumo35 Thanks for your reply.

Is version 6.8 really that bad? 6.8.17, in particular, was released only 3 months ago.

We have some other pipelines on 6.8 in the same environment, and we cannot go to version 7 yet.

[grumo35]

Oh i see i didnt pay attention to tailing subversion numbers, there is still 6.8.20 out there but i'm sure the bug i was referencing too came from olders versions or is patched in major 7.0 ...

I really dont know what could cause this issue