Intermittent metric registration error: worker_millis_per_event

Elastic Stack Logstash
1

Issue

Intermittent warning during pipeline startup. The worker_millis_per_event metric fails to register but pipeline functions normally.

[org.logstash.execution.AbstractPipelineExt] Metric registration error: `worker_millis_per_event` could not be registered in namespace `[:stats, :pipelines, :logstash, :plugins, :filters, :c13c16377fd9a15e6e691284665f2da3494b42273b1fc6e37b6330fff12eb5ab, :flow]`

Observed Behavior

  • Not consistently reproducible - Occurs intermittently during startup

  • No functional impact - Pipeline processes events normally

  • Metrics not available - Pipeline stats are empty from endpoint localhost:9600/_node/stats/pipelines
    {
    "host": "log-transformer",
    "version": "9.0.3",
    "http_address": "0.0.0.0:9600",
    "id": "5fbd8fxxxxxxxxxxxx5602",
    "name": "log-transformer",
    "ephemeral_id": "36axxxxxxxxxxxxxxxxx2348a7",
    "snapshot": false,
    "status": "unknown",
    "pipeline": {
    "workers": 2,
    "batch_size": 2048,
    "batch_delay": 50
    },
    "pipelines": {}
    }

  • Self-resolving: Logstash process restart with same configuration succeeds without error. No warning message and stats are also available in /_node/stats/pipelines endpoint.

Versions

  • Logstash: 9.0.3
  • JRuby: 9.4.9.0
  • JVM: OpenJDK17

Questions

  1. Is this a known race condition during pipeline initialization?

  2. Any recommended configuration to prevent this?

Logs

Full startup logs attached - Logstash Logs - Metrics registration error · GitHub

2

Your LS conf is OK. Listeners have been started correctly. Have you set legacy monitoring settings?

xpack.monitoring.allow_legacy_collection: true
xpack.monitoring.enabled: true

Do you have the LS_JAVA_HOME environment variable?
Have you installed JDK or you are using LS JDK bundled?

3

Thanks for responding. PFB details.

No, legacy monitoring is not set. PFB contents from logstash.yaml file. Hope those two settings are set to false by default.

api.http.host: "0.0.0.0"
api.http.port: 9600
log.level: "info"
pipeline.workers: 2
pipeline.batch.size: 2048
pipeline.batch.delay: 50
path.logs: /opt/logstash/resource
pipeline.ecs_compatibility: disabled

LS_JAVA_HOME is also not set. Logstash using default Java path which is /usr/bin/java

Installed JDK is used and not the Logstash bundled one.

Let me know if you need more details.

4

Can you try at least add: xpack.monitoring.allow_legacy_collection: true

You can check the line which raise an error. Not sure what's happening here because of Java. Long time ago I had a case, which was resolved by installing Oracle JDK not OpenJDK.

Anyone other had similar issues? Badger, Leandro, any suggestions?

5

I don't know has been this solved, a few more tips:

  • Set LS_JAVA_PATH. Check this topic about Java path issues.
  • Review jvm.options, log4j2.properties and JVM settings. Do you have upgraded LS version? Like from older 7.x or something? Check this for any case.
  • Maybe to try ​TAR.GZ version without docker, if is possible, at least temporarily.