# Data not appearing in elasticsearch after being received by APM server **URL:** \ **Category:** APM\ **Tags:** server\ **Created:** [February 15, 2023, 11:23pm UTC](https://discuss.elastic.co/t/data-not-appearing-in-elasticsearch-after-being-received-by-apm-server/325667 "2023-02-15T23:23:37Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Marko\_Todoric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marko_todoric/32/52975_2.png) [@Marko\_Todoric](https://discuss.elastic.co/u/Marko_Todoric)\ **Post date:** [February 15, 2023, 11:23pm UTC](https://discuss.elastic.co/t/data-not-appearing-in-elasticsearch-after-being-received-by-apm-server/325667/1 "2023-02-15T23:23:38Z")
**Kibana version** : 8.6.1 **Elasticsearch version** : 8.6.1 **APM Server version** : 8.6.1 **APM Agent language and version** : Java 8 (Elastic APM Agent 1.35 and 1.36) **Browser version** : Firefox 102.0 (64-bit) **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Elasticsearch: apt repo, elastic-agent: standalone **Fresh install or upgraded from other version?** Elasticsearch, APM Server (within fleet) upgraded all together from 8.5.3 to 8.6.1 **Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. I have nginx in front of APM servers **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: **Steps to reproduce** : 1. After upgrade from 8.5.3 to 8.6.1, data from **one** specific host (that belongs to 1 group of APM Servers) are no longer visible in Kibana, alert thrown `Document count reported no data in the last 1m for xxx` **Errors in browser console (if relevant)**: Not relevant **Provide logs and/or server output (if relevant)**: Nginx logs shows data being received ```auto xx.xx.xx.xx - - [16/Feb/2023:00:16:20 +0100] "POST /config/v1/agents HTTP/1.1" 304 0 "-" "apm-agent-java/1.36.0 (service-name)" xx.xx.xx.xx - - [16/Feb/2023:00:16:28 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)" xx.xx.xx.xx - - [16/Feb/2023:00:16:41 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)" xx.xx.xx.xx - - [16/Feb/2023:00:16:50 +0100] "POST /config/v1/agents HTTP/1.1" 304 0 "-" "apm-agent-java/1.36.0 (service-name)" xx.xx.xx.xx - - [16/Feb/2023:00:16:52 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)" xx.xx.xx.xx - - [16/Feb/2023:00:17:02 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)" ``` When looking at the elastic agent logs, data is received by APM server ```auto {"log.level":"info","@timestamp":"2023-02-15T23:16:28.711Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.origin":{"file.line":61,"file.name":"middleware/log_middleware.go"},"url.original":"/intake/v2/events","source.address":"xx.xx.xx.xx","http.request.id":"50d42347-fb2e-4c5f-8710-5586a4db205b","http.request.method":"POST","event.duration":33198072,"log.logger":"request","service.name":"apm-server","user_agent.original":"apm-agent-java/1.36.0 (service-name)","http.response.status_code":202,"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-02-15T23:16:52.105Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"url.original":"/intake/v2/events","user_agent.original":"apm-agent-java/1.36.0 (service-name)","log.origin":{"file.line":61,"file.name":"middleware/log_middleware.go"},"http.request.method":"POST","http.request.id":"7c67c7ea-4295-41d4-9601-3d6c8199a898","event.duration":18733716,"ecs.version":"1.6.0","http.response.status_code":202,"log.logger":"request","service.name":"apm-server","source.address":"xx.xx.xx.xx","ecs.version":"1.6.0"} ``` However when browsing Kibana, i see no data from that agent. And also, since apm server that agent is sending data to is having specific namespace, when looking at data stream, example " traces-apm-apm\_central" (apm\_central) being the namespace, i see it was last updated February 15th, 2023 3:51:26 PM (around the time upgrade of fleet agent was running). What happened and how to troubleshoot this , please? ---
**Author:** ![Marko\_Todoric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marko_todoric/32/52975_2.png) [@Marko\_Todoric](https://discuss.elastic.co/u/Marko_Todoric)\ **Post date:** [February 15, 2023, 11:54pm UTC](https://discuss.elastic.co/t/data-not-appearing-in-elasticsearch-after-being-received-by-apm-server/325667/2 "2023-02-15T23:54:42Z")
Ok, now this is weird. Also found log stating following: ```auto [elastic_agent.apm_server][error] failed to index event (fail_processor_exception): fail_processor_exception: Document produced by APM Server v8.6.1, which is newer than the installed APM integration (v8.5.0-preview-1663692203). The APM integration must be upgraded. ``` However APM integration is updated as well... ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/4/64fbb99e6603287673e3e67ac6b124d457eb986d.png) Elastic agent was restarted on all 3 hosts belonging to this policy, still the same... ---
**Author:** ![Marko\_Todoric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marko_todoric/32/52975_2.png) [@Marko\_Todoric](https://discuss.elastic.co/u/Marko_Todoric)\ **Post date:** [February 18, 2023, 10:00pm UTC](https://discuss.elastic.co/t/data-not-appearing-in-elasticsearch-after-being-received-by-apm-server/325667/3 "2023-02-18T22:00:36Z")
Temporary fix: Reinstall APM Integration. However it's only temporary for next 24 hours. After that issue happens again, then i have to reinstall APM Integration again. It's like this for few days now. Anyone has any idea how to troubleshoot this further? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2023, 6:01pm UTC](https://discuss.elastic.co/t/data-not-appearing-in-elasticsearch-after-being-received-by-apm-server/325667/4 "2023-03-11T18:01:20Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.