Intermittent metric registration error: worker_millis_per_event

Elastic Stack Logstash
1

Issue

Intermittent warning during pipeline startup. The worker_millis_per_event metric fails to register but pipeline functions normally.

[org.logstash.execution.AbstractPipelineExt] Metric registration error: `worker_millis_per_event` could not be registered in namespace `[:stats, :pipelines, :logstash, :plugins, :filters, :c13c16377fd9a15e6e691284665f2da3494b42273b1fc6e37b6330fff12eb5ab, :flow]`

Observed Behavior

  • Not consistently reproducible - Occurs intermittently during startup

  • No functional impact - Pipeline processes events normally

  • Metrics not available - Pipeline stats are empty from endpoint localhost:9600/_node/stats/pipelines
    {
    "host": "log-transformer",
    "version": "9.0.3",
    "http_address": "0.0.0.0:9600",
    "id": "5fbd8fxxxxxxxxxxxx5602",
    "name": "log-transformer",
    "ephemeral_id": "36axxxxxxxxxxxxxxxxx2348a7",
    "snapshot": false,
    "status": "unknown",
    "pipeline": {
    "workers": 2,
    "batch_size": 2048,
    "batch_delay": 50
    },
    "pipelines": {}
    }

  • Self-resolving: Logstash process restart with same configuration succeeds without error. No warning message and stats are also available in /_node/stats/pipelines endpoint.

Versions

  • Logstash: 9.0.3
  • JRuby: 9.4.9.0
  • JVM: OpenJDK17

Questions

  1. Is this a known race condition during pipeline initialization?

  2. Any recommended configuration to prevent this?

Logs

Full startup logs attached - Logstash Logs - Metrics registration error · GitHub

2

Your LS conf is OK. Listeners have been started correctly. Have you set legacy monitoring settings?

xpack.monitoring.allow_legacy_collection: true
xpack.monitoring.enabled: true

Do you have the LS_JAVA_HOME environment variable?
Have you installed JDK or you are using LS JDK bundled?

3

Thanks for responding. PFB details.

No, legacy monitoring is not set. PFB contents from logstash.yaml file. Hope those two settings are set to false by default.

api.http.host: "0.0.0.0"
api.http.port: 9600
log.level: "info"
pipeline.workers: 2
pipeline.batch.size: 2048
pipeline.batch.delay: 50
path.logs: /opt/logstash/resource
pipeline.ecs_compatibility: disabled

LS_JAVA_HOME is also not set. Logstash using default Java path which is /usr/bin/java

Installed JDK is used and not the Logstash bundled one.

Let me know if you need more details.

4

Can you try at least add: xpack.monitoring.allow_legacy_collection: true

You can check the line which raise an error. Not sure what's happening here because of Java. Long time ago I had a case, which was resolved by installing Oracle JDK not OpenJDK.

Anyone other had similar issues? Badger, Leandro, any suggestions?

5

I don't know has been this solved, a few more tips:

  • Set LS_JAVA_PATH. Check this topic about Java path issues.
  • Review jvm.options, log4j2.properties and JVM settings. Do you have upgraded LS version? Like from older 7.x or something? Check this for any case.
  • Maybe to try ​TAR.GZ version without docker, if is possible, at least temporarily.
6

Hi @Rios

Sorry for the delay here.

Currently, testing Logstash with xpack.monitoring.allow_legacy_collection: true parameter. Will share the observations shortly.

Regarding log4j2.properties, we are not using any properties related to routing/rolling/json_rolling. So those changes are not relevant to us at least. Snippet from our log4j2.properties file.

monitorInterval = 1
appender.console.type = Console
appender.console.name = plain_console
appender.console.layout.type = PatternLayout

Thanks,
Mariya

7

@Rios Tried adding above mentioned properties in logstash.yml file but it’s not supported since we are using logstash-oss version.

Your settings are invalid. Reason: Setting "xpack.monitoring.allow_legacy_collection" doesn't exist. Please check if you haven't made a typo.

Kindly suggest here.

8

Yes you have abs. right, OSS doesn't support x-pack.
However, I found info that APIs should work, but cannot find an official confirmation.

Regarding log4j2.properties params, this can be related if LS was upgraded from older version and miss paramters. In case there was no upgrade or nobody has been changed params, leave aside.

Again, according to the s.code on GitHub, an issue might be related to Java. Have you tried to set LS_JAVA_HOME?

Have you tried with .gz version? Just for test purpose. Extract, test, later can be removed.