Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response

Elastic Stack Beats
,
1

Hi,

I have just upgraded our ELK-stack to 8.3.3 and I am migrating the monitoring to use Metricbeat. Elasticsearch and Kibana works without any issues. However for Logstash I'm seeing the following in the logs

{
    "log.level": "error",
    "@timestamp": "2022-08-09T12:25:32.187+0200",
    "log.origin": {
        "file.name": "module/wrapper.go",
        "file.line": 256
    },
    "message": "Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response",
    "service.name": "metricbeat",
    "ecs.version": "1.6.0"
}

There are two similar threads but they are both closed due to inactivity:

Unfortunately I cannot find a solution to my error from those threads. I have done the following:

  • Verified that the Logstash API endpoints return an id field in the response
  • Verified that the target index is a data stream
  • Verified that Logstash monitoring is visible in Kibana Stack Monitoring view
  • Enabled debug logging for Metricbeat to get more details

Some details about our setup

  • Everything is running on version 8.3.3
  • We are running without any security, so no auth and no tls
  • We are running Logstash with xpack.monitoring.enabled and not monitoring.enabled because I got a license check error without the xpack.monitoring.hosts configuration and it is not possible to run with both xpack.monitoring and monitoring simultaneously
  • We are running Metricbeat with all the module configured inlined in the main configuration file (i.e. without running metricbeat modules enable logstash-xpack)

Metricbeat configuration:

logging:
  level: info
  to_files: true
  files:
    path: ${path.logs}
    name: metricbeat.log
    keepfiles: 7
    permissions: 0640
  metrics:
    enabled: false

metricbeat:
  modules:
    - module: elasticsearch
      xpack.enabled: true
      period: 10s
      hosts:
        - http://7018.company.net:9200
      scope: node
    - module: logstash
      xpack.enabled: true
      period: 10s
      hosts:
        - http://localhost:9600
    - module: kibana
      xpack.enabled: true
      period: 10s
      hosts:
        - http://169.254.1.1:48101

name: 7018.company.net

output:
  elasticsearch:
    hosts:
      - http://7018.company.net:9200

path:
  home: /opt/company/elk-monitoring-test/install/current
  config: /opt/company/elk-monitoring-test/config
  data: /opt/company/elk-monitoring-test/data
  logs: /opt/company/elk-monitoring-test/logs

The error persists and I am not really sure what to try next and would appreciate some input!

2

@andreastoom

Hi

I am facing the same issue.

I have also found the following post on github.

I don't understand the details of this post, but I see "v8.4.0" in the label, so I assume that a update to this issue will be released in the future.

3

That looks promising - I wonder if there are nightly builds available for download ? or if I should try to build the main branch locally and see if the error is fixed!

4

@andreastoom 8.4 is out! , could you fix it? Thank you

5

Hi,

Just tried the new version and I am no longer seeing the original error so it seem to have been fixed!

However, I am seeing the log line below instead but I don't have time to troubleshoot the details. Could be because I was running Metricbeat 8.4.0 against Logstash 8.3.3.

metricbeat.log-20220830.ndjson:{"log.level":"warn","@timestamp":"2022-08-30T06:31:30.712+0200","log.logger":"logstash.node_stats","log.origin":{"file.name":"node_stats/data.go","file.line":189},"message":"Pipeline document was discarded due to missing properties. This can happen when the Logstash node stats API is polled before the pipeline setup has completed. Pipeline ID: .monitoring-logstash","service.name":"metricbeat","ecs.version":"1.6.0"}
6

@andreastoom
@Gustavo_Llermaly

I tried the new version and I am no longer seeing the original error too.

Also, logstash 8.4 does not seem to produce the warning message that andreastoom reported.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.