Kibana Observability Metrics - Issue

[Ravi342883 (RAVINDRAN SHANMUGAM)]

Hello Elastic Community Members,

Currently, I'm facing issue in loading the metrics (auto populated graphs) over the metricbeat data.

I tried 2 scenarios here to figure out this issue, let me brief that in detail.

Scenario1:

1. Installed metricbeat(V 7.11.X) in server to send data directly to elastic search
2. Metricbeat -> Elastic Search
3. In this setup, we are able to view the data under Kibana -> Observability -> Metrics. Able to see the default metric graphs in Inventory section.
4. Refer screenshot - scenario1.jpg

Scenario2:

1. Installed metricbeat (V 7.11.X) in server to send data to AWS MSK, then logstash reads data from AWS MSK & outputs into Elasticsearch.
2. Metricbeat -> AWS MSK -> Logstash -> Elastic Search
3. In this setup, we are not able to view the data under Kibana -> Observability -> Metrics. Able to see the below error when we click the Inventory section.

Request URL: https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.aws.elastic-cloud.com:9243/api/metrics/snapshot
Response:
{"statusCode":500,"error":"Internal Server Error","message":"[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}
Refer Scenario2_error.jpg attached.

4. Refer screenshot - scenario2.jpg

Need your assistance to fix the issue in above scenario2. I need to get the default metrics graphs under Observability, when we ship data via route Metricbeat -> AWS MSK -> Logstash -> Elastic Search.

Thanks,
Ravi
+44 7404322141

[weltenwort (Felix Stürmer)]

Hi @Ravi342883,

regardless of what the exact ingestion pathway looks like, it is important that the correct index templates are installed before any data are ingested. Otherwise the index mappings will be guessed dynamically by Elasticsearch and lead to the errors you report. The canonical way of installing the index templates is to stop all metricbeat shippers, remove all incorrectly mapped indices and run metricbeat setup to install the index templates. Since you're using a customized ingestion pathway you then have to make sure that the index names used in the final ingestion step match the pattern defined in the index template (or adapt that pattern).

Hope that helps a bit.

[Ravi342883 (RAVINDRAN SHANMUGAM)]

Hello @weltenwort - Thanks for your response.

I tired the same, used the index/template generated in scenario1 into scenario2. It worked - I could see the data/graphs under observability section.

Thanks,
Ravi