# Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response **URL:** \ **Category:** Beats\ **Tags:** elastic-stack-monitoring, metricbeat\ **Created:** [August 9, 2022, 11:25am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717 "2022-08-09T11:25:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![andreastoom](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@andreastoom](https://discuss.elastic.co/u/andreastoom)\ **Post date:** [August 9, 2022, 11:25am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/1 "2022-08-09T11:25:33Z")
Hi, I have just upgraded our ELK-stack to 8.3.3 and I am migrating the monitoring to use Metricbeat. Elasticsearch and Kibana works without any issues. However for Logstash I'm seeing the following in the logs ```auto { "log.level": "error", "@timestamp": "2022-08-09T12:25:32.187+0200", "log.origin": { "file.name": "module/wrapper.go", "file.line": 256 }, "message": "Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response", "service.name": "metricbeat", "ecs.version": "1.6.0" } ``` There are two similar threads but they are both closed due to inactivity: - [Metricbeat Cluster Monitoring: logstash.node: Could not find field 'id' in Logstash API response](https://discuss.elastic.co/t/metricbeat-cluster-monitoring-logstash-node-could-not-find-field-id-in-logstash-api-response/299651) - [[metricbeat]Error fetching data for metricset logstash.node: Could not find field 'id' in Logstash API response](https://discuss.elastic.co/t/metricbeat-error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/299418) Unfortunately I cannot find a solution to my error from those threads. I have done the following: - Verified that the Logstash API endpoints return an `id` field in the response - Verified that the target index is a data stream - Verified that Logstash monitoring is visible in Kibana Stack Monitoring view - Enabled debug logging for Metricbeat to get more details Some details about our setup - Everything is running on version `8.3.3` - We are running without any security, so no auth and no tls - We are running Logstash with `xpack.monitoring.enabled` and not `monitoring.enabled` because I got a license check error without the `xpack.monitoring.hosts` configuration and it is not possible to run with both `xpack.monitoring` and `monitoring` simultaneously - We are running Metricbeat with all the module configured inlined in the main configuration file (i.e. without running `metricbeat modules enable logstash-xpack`) Metricbeat configuration: ```auto logging: level: info to_files: true files: path: ${path.logs} name: metricbeat.log keepfiles: 7 permissions: 0640 metrics: enabled: false metricbeat: modules: - module: elasticsearch xpack.enabled: true period: 10s hosts: - http://7018.company.net:9200 scope: node - module: logstash xpack.enabled: true period: 10s hosts: - http://localhost:9600 - module: kibana xpack.enabled: true period: 10s hosts: - http://169.254.1.1:48101 name: 7018.company.net output: elasticsearch: hosts: - http://7018.company.net:9200 path: home: /opt/company/elk-monitoring-test/install/current config: /opt/company/elk-monitoring-test/config data: /opt/company/elk-monitoring-test/data logs: /opt/company/elk-monitoring-test/logs ``` The error persists and I am not really sure what to try next and would appreciate some input! ---
**Author:** ![Yos](https://avatars.discourse-cdn.com/v4/letter/y/ed8c4c/32.png) [@Yos](https://discuss.elastic.co/u/Yos)\ **Post date:** [August 10, 2022, 12:37am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/2 "2022-08-10T00:37:30Z")
@andreastoom Hi I am facing the same issue. I have also found the following post on github. > > > \## What does this PR do? > > This PR closes https://github.com/elastic/beats/issu…es/31739. > > This PR stops multiple errors being logged when using Logstash with multiple pipelines. > > ⚠️ \*\*Please see the fields / mapping inconsistency section I've added in this description, because whilst this PR fixes the logging issue, there are some questions.\*\* ⚠️ > > \## Why is it important? > > The Logs are noisy. > > \## Checklist > > \- \[x\] My code follows the style guidelines of this project (I think so, couldn't find a STYLEGUIDE file) > \- \[\] I have commented my code, particularly in hard-to-understand areas > \- \[\] I have made corresponding changes to the documentation > \- \[\] I have made corresponding change to the default configuration files > \- \[\] I have added tests that prove my fix is effective or that my feature works > \- \[\] I have added an entry in \`CHANGELOG.next.asciidoc\` or \`CHANGELOG-developer.next.asciidoc\`. > > \## Author's Checklist > > \## How to test this PR locally > > \- Follow the docs to setup your environment to build Beats (here: https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html#setting-up-dev-environment) > > \- Checkout this branch, and then\`cd metricbeat\`, and then \`mage build\`. > > \- Make a Metricbeat config file that contains the following (change the credentials): > > \`\`\`yml > http.enabled: true > metricbeat.modules: > - module: system > > - module: logstash > xpack.enabled: true > period: 10s > hosts: \["localhost:9600" \] > > output.elasticsearch: > hosts: \["localhost:9200" \] > username: "elastic" > password: "changeme" > \`\`\` > > \- Then run \`./metricbeat -e -c PATH\_TO\_YOUR\_METRICBEAT\_YML\_FILE\` > > \- \[Run Logstash with multiple pipelines\](https://github.com/elastic/kibana/blob/main/x-pack/plugins/monitoring/dev\_docs/how\_to/local\_setup.md#standalone-cluster). > > \`\`\` > docker run --name logstash \\ > --pull always --rm \\ > --hostname=logstash \\ > --publish=9600:9600 \\ > --volume="$(pwd)/x-pack/plugins/monitoring/dev\_docs/reference/logstash.yml:/usr/share/logstash/config/logstash.yml:ro" \\ > --volume="$(pwd)/x-pack/plugins/monitoring/dev\_docs/reference/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro" \\ > docker.elastic.co/logstash/logstash:master-SNAPSHOT > \`\`\` > > > You should \*\*not\*\* see excessive logging, e.g.: > > !\[Screenshot 2022-06-17 at 17 19 20\](https://user-images.githubusercontent.com/471693/174350731-511ad4f7-e8e3-4113-acaf-e8d656556993.png) > > > \## Related issues > > Closes https://github.com/elastic/beats/issues/31739. > > > \## Logs > > Excessive logging before: > > !\[Screenshot 2022-06-17 at 17 19 20\](https://user-images.githubusercontent.com/471693/174350731-511ad4f7-e8e3-4113-acaf-e8d656556993.png) > > \## Fields / mapping inconsistency > > It is noted in the issue that: > > \> The deletions seem to have been added in https://github.com/elastic/beats/pull/10350 presumably to avoid leaving fields in the document that don't comply with ECS > > and this seemed like a very reasonable assumption. As such I originally moved > > \`\`\`go > event.MetricSetFields.Update(fields) > \`\`\` > > below > > \`\`\`go > if err = commonFieldsMapping(&event, fields); err != nil { > return err > } > \`\`\` > > because it's \`commonFieldsMapping\` that calls \`fields.Delete()\`. This means things like \`host\` and \`version\` wouldn't end up on \`logstash.node\`. \[This also aligned with the documentation\](https://www.elastic.co/guide/en/beats/metricbeat/master/exported-fields-logstash.html). We can see there that \`logstash.node.host\` and co are supposed to be an alias. However, that's not the way the mappings seem to work. \[These are not aliased\](https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/monitoring-logstash-mb.json#L286). So having \`event.MetricSetFields.Update(fields)\` copy everything (the full set of fields) seems necessary. It doesn't align with the docs, but it aligns with the mappings (and most likely how solutions are accessing these fields). There definitely seems to be confusion here in expectation, unless I'm misunderstanding. > > \[I have also added a Gist here\](https://gist.github.com/Kerry350/278c337888b3e457e919c1ce4cdc5b69) with results produced by all three. Before changes, after changes with field deletion, and after changes with no field deletion. We can see that in terms of indexed data before and after changes with no field deletion match. > > As such, this PR just fixes the logging problem by making a new \`fields\` map each time, which \[aligns with this loop example\](https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/enrich/data.go#L88). > > I am not sure how we want to approach the fields and mappings. It seems that we should update the mappings to accurately reflect the docs regarding aliases, and \`id\`, \`host\`, \`version\` should no longer exist on \`logstash.node\`. I don't understand the details of this post, but I see "v8.4.0" in the label, so I assume that a update to this issue will be released in the future. ---
**Author:** ![andreastoom](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@andreastoom](https://discuss.elastic.co/u/andreastoom)\ **Post date:** [August 10, 2022, 4:28am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/3 "2022-08-10T04:28:04Z")
That looks promising - I wonder if there are nightly builds available for download ? or if I should try to build the main branch locally and see if the error is fixed! ---
**Author:** ![Gustavo\_Llermaly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gustavo_llermaly/32/92019_2.png) [@Gustavo\_Llermaly](https://discuss.elastic.co/u/Gustavo_Llermaly)\ **Post date:** [August 29, 2022, 12:32pm UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/4 "2022-08-29T12:32:28Z")
> [@andreastoom](#): > > Enabled debug logging for Metricbeat to get more details @andreastoom 8.4 is out! , could you fix it? Thank you ---
**Author:** ![andreastoom](https://avatars.discourse-cdn.com/v4/letter/a/7c8e57/32.png) [@andreastoom](https://discuss.elastic.co/u/andreastoom)\ **Post date:** [August 30, 2022, 4:34am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/5 "2022-08-30T04:34:22Z")
Hi, Just tried the new version and I am no longer seeing the original error so it seem to have been fixed! However, I am seeing the log line below instead but I don't have time to troubleshoot the details. Could be because I was running Metricbeat 8.4.0 against Logstash 8.3.3. ```auto metricbeat.log-20220830.ndjson:{"log.level":"warn","@timestamp":"2022-08-30T06:31:30.712+0200","log.logger":"logstash.node_stats","log.origin":{"file.name":"node_stats/data.go","file.line":189},"message":"Pipeline document was discarded due to missing properties. This can happen when the Logstash node stats API is polled before the pipeline setup has completed. Pipeline ID: .monitoring-logstash","service.name":"metricbeat","ecs.version":"1.6.0"} ``` ---
**Author:** ![Yos](https://avatars.discourse-cdn.com/v4/letter/y/ed8c4c/32.png) [@Yos](https://discuss.elastic.co/u/Yos)\ **Post date:** [September 6, 2022, 4:29am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/6 "2022-09-06T04:29:01Z")
@andreastoom @Gustavo_Llermaly I tried the new version and I am no longer seeing the original error too. Also, logstash 8.4 does not seem to produce the warning message that andreastoom reported. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2022, 6:29am UTC](https://discuss.elastic.co/t/error-fetching-data-for-metricset-logstash-node-could-not-find-field-id-in-logstash-api-response/311717/7 "2022-10-04T06:29:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.