# Kibana APM not showing any data for the last two weeks
**URL:** https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291
**Category:** APM
**Tags:** server, open-telemetry
**Created:** [August 21, 2023, 5:50pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291 "2023-08-21T17:50:12Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@ELLIOTTCABLE](https://discuss.elastic.co/u/ELLIOTTCABLE)
#### Post date: [August 21, 2023, 5:50pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/1 "2023-08-21T17:50:12Z")
> **Kibana version** : v8.8.1
>
> **Elasticsearch version** : v8.8.1
>
> **APM Server version** : v8.8.0
>
> **Browser version** : FireFox v **116.0**.3; Chrome v **115.0**.5790.170
>
> **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: N/A
>
> **Fresh install or upgraded from other version?** : N/A
>
> **Is there anything special in your setup?** Logstash is being used (separate, not yet integrated with OpenTelemetry/APM stuff); A load-balancer is in front of the APM servers, I think.
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:
We've just recently started our OpenTelemetry journey; this hasn't all been completely set up yet; but I _had_ OpenTelemetry-reported traces, via `apm-server`, showing up in Kibana for a month or two as I worked on things.
Unfortunately, starting a couple weeks ago, OpenTelemetry data / traces suddenly stopped appearing in the "APM" section of Kibana:
> > **Screenshot of missing data, filtered to 'past two weeks':**
> >
> > 
… more content in later posts, because this forum has _ridiculous_ spam-prevention limits for an official support-forum (one image? no links?? seriously??) …
---
### Author:  [@ELLIOTTCABLE](https://discuss.elastic.co/u/ELLIOTTCABLE)
#### Post date: [August 21, 2023, 5:51pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/2 "2023-08-21T17:51:07Z")
If I search farther back, older data is visible:
> > **Screenshot of data present, 21 days ago:**
> >
> > 
---
### Author:  [@ELLIOTTCABLE](https://discuss.elastic.co/u/ELLIOTTCABLE)
#### Post date: [August 21, 2023, 5:52pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/3 "2023-08-21T17:52:07Z")
The data _exists_, as I can track down individual events in the "Discover" view:
> > **Screenshot of recent data visible in the 'Discover' view:**
> >
> > 
---
### Author:  [@ELLIOTTCABLE](https://discuss.elastic.co/u/ELLIOTTCABLE)
#### Post date: [August 21, 2023, 5:53pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/4 "2023-08-21T17:53:08Z")
One potential culprit I've found, digging around in Kibana settings, is this error:
> **Mapping conflict** : 3 fields are defined as several types (string, integer, etc) across the indices that match this pattern. You may still be able to use these conflict fields in parts of Kibana, but they will be unavailable for functions that require Kibana to know their type. Correcting this issue will require reindexing your data.
>
> > **Screenshot of error message and conflicted fields:**
> >
> > 
The conflicted fields are these:
- `event.success_count`: `byte`, `aggregate_metric_double`, `long`, `object`
- `transaction.duration.histogram`: `histogram`, `object`
- `transaction.duration.summary`: `aggregate_metric_double`, `object`
I did find [this help article](https://www.elastic.co/guide/en/kibana/current/troubleshooting.html#no-apm-data-found) and this blog post (dev dot sobeslavsky dot net slash kibana-how-to-solve-mapping-conflict); but before I can even start to follow those instructions (which I think seem to amount to 'set a fixed type for the fields in an "index template"), because I don't know which of those types each one _should_ be.
Relatedly: How on earth did multiple types get set for these values? Aren't these part of APM? They don't seem to be anything under our control. How can I prevent this occurring in the future, when OpenTelemetry becomes critical to our investigation infrastructure, instead of just an experimental toy?
---
### Author:  [@ELLIOTTCABLE](https://discuss.elastic.co/u/ELLIOTTCABLE)
#### Post date: [August 21, 2023, 8:38pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/5 "2023-08-21T20:38:55Z")
Okay, so I think something must have gone wrong during an Elasticsearch upgrade — the fields that are conflicted are precisely those that [have breaking changes in APM](https://www.elastic.co/guide/en/apm/guide/8.9/apm-breaking.html#breaking-changes-8.7). I found no mention of any errors reported by our infra people, though, so I'm not sure how or when this happened …
The indices created before July 22nd, and the new ones created after Aug 1st, correctly(?) have `event.success_count` (for example) declared as a `"aggregate_metric_double"` … but in that time-window, indices were getting created with `event.success_count` as a `"byte"`?
```json
GET .ds-*apm*-2023.07.22-*/_mapping/field/event.success_count
// produces:
{
".ds-metrics-apm.transaction.10m-default-2023.07.22-000004": {
"mappings": {
"event.success_count": {
"full_name": "event.success_count",
"mapping": {
"success_count": {
"type": "aggregate_metric_double",
// ...
}}}}}}
GET .ds-*apm*-2023.07.23-*/_mapping/field/event.success_count
// produces:
{
".ds-traces-apm-default-2023.07.23-000454": {
"mappings": {
"event.success_count": {
"full_name": "event.success_count",
"mapping": {
"success_count": {
"type": "byte",
// ...
}}}}}}
GET .ds-*apm*-2023.08.01-*/_mapping/field/event.success_count
// produces:
{
".ds-metrics-apm.transaction.1m-default-2023.08.01-000019": {
"mappings": {
"event.success_count": {
"full_name": "event.success_count",
"mapping": {
"success_count": {
"type": "aggregate_metric_double",
"type": "byte",
// ...
}}}}}}
```
(How could this have happened?)
At least one option, I think, is to wipe out all the traces data entirely; but that feels nuclear. It looks like I _should_, at least, be able to ["use a reindex to change mappings" on the "data stream"](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/modify-data-streams.html#data-streams-use-reindex-to-change-mappings-settings); but I'm not sure if I can _tell APM_ to use the new, reindexed data-stream afterwards …
---
### Author:  [@xeraa](https://discuss.elastic.co/u/xeraa)
#### Post date: [August 21, 2023, 8:42pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/6 "2023-08-21T20:42:24Z")
I took a quick look at `event.success_count` and I have some pointers but no proper answer yet: I see a [breaking change in 8.7](https://www.elastic.co/guide/en/apm/guide/current/release-notes-8.7.html#_breaking_changes_2). Did you ever upgrade components or were they always on this version? Not a great answer but potentially a clue where this went wrong.
And I tried to find the right datatype but I'm getting mixed signals on 8.8:
- `aggregate_metric_double` from [https://github.com/elastic/apm-server/blob/8.8/apmpackage/apm/data\_stream/transaction\_interval\_metrics/manifest.yml#L30](https://github.com/elastic/apm-server/blob/8.8/apmpackage/apm/data_stream/transaction_interval_metrics/manifest.yml#L30)
- `byte` from [https://github.com/elastic/apm-server/blob/8.8/apmpackage/apm/data\_stream/traces/fields/fields.yml#L26](https://github.com/elastic/apm-server/blob/8.8/apmpackage/apm/data_stream/traces/fields/fields.yml#L26)
- `long` might be the default dynamic mapping if the field is unknown in the index template (maybe a version conflict)
- `object` would be if you have another subfield (whatever that might be in your case)
---
### Author:  [@ELLIOTTCABLE](https://discuss.elastic.co/u/ELLIOTTCABLE)
#### Post date: [August 21, 2023, 8:49pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/7 "2023-08-21T20:49:26Z")
(also worth noting — before the 22nd, only the `metrics-apm.transaction` indices had the `event.success_count` mapping, it was completely absent for other indices; between the 22nd and 1st, _every single APM-related index(!)_ has the `type: "byte"` mapping for `event.success_count`; and finally, post-1st, although `metrics-apm.transaction` is back to being the correct type … all of our other APM-related indices now report empty `{}` for that key?
```json
GET .ds-*apm*-2023.08.01-*/_mapping/field/event.success_count
// produces:
{
// ...
".ds-traces-apm-default-2023.08.01-000810": {
"mappings": {}
},
".ds-traces-apm-default-2023.08.01-000799": {
"mappings": {}
},
".ds-traces-apm-default-2023.08.01-000798": {
"mappings": {}
},
".ds-metrics-apm.transaction.1m-default-2023.08.01-000019": {
"mappings": {
"event.success_count": {
"full_name": "event.success_count",
"mapping": {
"success_count": {
"type": "aggregate_metric_double",
"metrics": [
"sum",
"value_count"
],
"default_metric": "sum"
}
}
}
}
},
".ds-traces-apm-default-2023.08.01-000809": {
"mappings": {}
},
".ds-traces-apm-default-2023.08.01-000808": {
"mappings": {}
},
".ds-traces-apm-default-2023.08.01-000807": {
"mappings": {}
},
// ...
}
```
---
### Author:  [@xeraa](https://discuss.elastic.co/u/xeraa)
#### Post date: [August 22, 2023, 9:16pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/8 "2023-08-22T21:16:20Z")
Like I said on Slack: That looks like some upgrade issue or a very weird combination of things. Depending on how important the data is, you could either delete the indices, reindex to fix the mapping, or maybe fix it at query-time through runtime fields.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 19, 2023, 9:17pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-any-data-for-the-last-two-weeks/341291/9 "2023-09-19T21:17:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.