Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response

andreastoom · August 9, 2022, 11:25am

Hi,

I have just upgraded our ELK-stack to 8.3.3 and I am migrating the monitoring to use Metricbeat. Elasticsearch and Kibana works without any issues. However for Logstash I'm seeing the following in the logs

{
    "log.level": "error",
    "@timestamp": "2022-08-09T12:25:32.187+0200",
    "log.origin": {
        "file.name": "module/wrapper.go",
        "file.line": 256
    },
    "message": "Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response",
    "service.name": "metricbeat",
    "ecs.version": "1.6.0"
}

There are two similar threads but they are both closed due to inactivity:

Unfortunately I cannot find a solution to my error from those threads. I have done the following:

Verified that the Logstash API endpoints return an id field in the response
Verified that the target index is a data stream
Verified that Logstash monitoring is visible in Kibana Stack Monitoring view
Enabled debug logging for Metricbeat to get more details

Some details about our setup

Everything is running on version 8.3.3
We are running without any security, so no auth and no tls
We are running Logstash with xpack.monitoring.enabled and not monitoring.enabled because I got a license check error without the xpack.monitoring.hosts configuration and it is not possible to run with both xpack.monitoring and monitoring simultaneously
We are running Metricbeat with all the module configured inlined in the main configuration file (i.e. without running metricbeat modules enable logstash-xpack)

Metricbeat configuration:

logging:
  level: info
  to_files: true
  files:
    path: ${path.logs}
    name: metricbeat.log
    keepfiles: 7
    permissions: 0640
  metrics:
    enabled: false

metricbeat:
  modules:
    - module: elasticsearch
      xpack.enabled: true
      period: 10s
      hosts:
        - http://7018.company.net:9200
      scope: node
    - module: logstash
      xpack.enabled: true
      period: 10s
      hosts:
        - http://localhost:9600
    - module: kibana
      xpack.enabled: true
      period: 10s
      hosts:
        - http://169.254.1.1:48101

name: 7018.company.net

output:
  elasticsearch:
    hosts:
      - http://7018.company.net:9200

path:
  home: /opt/company/elk-monitoring-test/install/current
  config: /opt/company/elk-monitoring-test/config
  data: /opt/company/elk-monitoring-test/data
  logs: /opt/company/elk-monitoring-test/logs

The error persists and I am not really sure what to try next and would appreciate some input!

Yos · August 10, 2022, 12:37am

@andreastoom

Hi

I am facing the same issue.

I have also found the following post on github.

github.com/elastic/beats

[Metricbeat] Stop continual error with multiple Logstash pipelines

elastic:main ← Kerry350:31739-stop-continual-error-with-multiple-pipelines

opened 05:57PM - 17 Jun 22 UTC

Kerry350

+5 -5

## What does this PR do? This PR closes https://github.com/elastic/beats/issu…es/31739. This PR stops multiple errors being logged when using Logstash with multiple pipelines. ⚠️ **Please see the fields / mapping inconsistency section I've added in this description, because whilst this PR fixes the logging issue, there are some questions.** ⚠️ ## Why is it important? The Logs are noisy. ## Checklist - [x] My code follows the style guidelines of this project (I think so, couldn't find a STYLEGUIDE file) - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] I have made corresponding change to the default configuration files - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`. ## Author's Checklist ## How to test this PR locally - Follow the docs to setup your environment to build Beats (here: https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html#setting-up-dev-environment) - Checkout this branch, and then`cd metricbeat`, and then `mage build`. - Make a Metricbeat config file that contains the following (change the credentials): ```yml http.enabled: true metricbeat.modules: - module: system - module: logstash xpack.enabled: true period: 10s hosts: [ "localhost:9600" ] output.elasticsearch: hosts: [ "localhost:9200" ] username: "elastic" password: "changeme" ``` - Then run `./metricbeat -e -c PATH_TO_YOUR_METRICBEAT_YML_FILE` - [Run Logstash with multiple pipelines](https://github.com/elastic/kibana/blob/main/x-pack/plugins/monitoring/dev_docs/how_to/local_setup.md#standalone-cluster). ``` docker run --name logstash \ --pull always --rm \ --hostname=logstash \ --publish=9600:9600 \ --volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/logstash.yml:/usr/share/logstash/config/logstash.yml:ro" \ --volume="$(pwd)/x-pack/plugins/monitoring/dev_docs/reference/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro" \ docker.elastic.co/logstash/logstash:master-SNAPSHOT ``` You should **not** see excessive logging, e.g.: ![Screenshot 2022-06-17 at 17 19 20](https://user-images.githubusercontent.com/471693/174350731-511ad4f7-e8e3-4113-acaf-e8d656556993.png) ## Related issues Closes https://github.com/elastic/beats/issues/31739. ## Logs Excessive logging before: ![Screenshot 2022-06-17 at 17 19 20](https://user-images.githubusercontent.com/471693/174350731-511ad4f7-e8e3-4113-acaf-e8d656556993.png) ## Fields / mapping inconsistency It is noted in the issue that: > The deletions seem to have been added in https://github.com/elastic/beats/pull/10350 presumably to avoid leaving fields in the document that don't comply with ECS and this seemed like a very reasonable assumption. As such I originally moved ```go event.MetricSetFields.Update(fields) ``` below ```go if err = commonFieldsMapping(&event, fields); err != nil { return err } ``` because it's `commonFieldsMapping` that calls `fields.Delete()`. This means things like `host` and `version` wouldn't end up on `logstash.node`. [This also aligned with the documentation](https://www.elastic.co/guide/en/beats/metricbeat/master/exported-fields-logstash.html). We can see there that `logstash.node.host` and co are supposed to be an alias. However, that's not the way the mappings seem to work. [These are not aliased](https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/monitoring-logstash-mb.json#L286). So having `event.MetricSetFields.Update(fields)` copy everything (the full set of fields) seems necessary. It doesn't align with the docs, but it aligns with the mappings (and most likely how solutions are accessing these fields). There definitely seems to be confusion here in expectation, unless I'm misunderstanding. [I have also added a Gist here](https://gist.github.com/Kerry350/278c337888b3e457e919c1ce4cdc5b69) with results produced by all three. Before changes, after changes with field deletion, and after changes with no field deletion. We can see that in terms of indexed data before and after changes with no field deletion match. As such, this PR just fixes the logging problem by making a new `fields` map each time, which [aligns with this loop example](https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/enrich/data.go#L88). I am not sure how we want to approach the fields and mappings. It seems that we should update the mappings to accurately reflect the docs regarding aliases, and `id`, `host`, `version` should no longer exist on `logstash.node`.

I don't understand the details of this post, but I see "v8.4.0" in the label, so I assume that a update to this issue will be released in the future.

andreastoom · August 10, 2022, 4:28am

That looks promising - I wonder if there are nightly builds available for download ? or if I should try to build the main branch locally and see if the error is fixed!

Gustavo_Llermaly · August 29, 2022, 12:32pm

@andreastoom 8.4 is out! , could you fix it? Thank you

andreastoom · August 30, 2022, 4:34am

Hi,

Just tried the new version and I am no longer seeing the original error so it seem to have been fixed!

However, I am seeing the log line below instead but I don't have time to troubleshoot the details. Could be because I was running Metricbeat 8.4.0 against Logstash 8.3.3.

metricbeat.log-20220830.ndjson:{"log.level":"warn","@timestamp":"2022-08-30T06:31:30.712+0200","log.logger":"logstash.node_stats","log.origin":{"file.name":"node_stats/data.go","file.line":189},"message":"Pipeline document was discarded due to missing properties. This can happen when the Logstash node stats API is polled before the pipeline setup has completed. Pipeline ID: .monitoring-logstash","service.name":"metricbeat","ecs.version":"1.6.0"}

Yos · September 6, 2022, 4:29am

@andreastoom
@Gustavo_Llermaly

I tried the new version and I am no longer seeing the original error too.

Also, logstash 8.4 does not seem to produce the warning message that andreastoom reported.

system · October 4, 2022, 6:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.