[Metrics] Error when trying to fetch data

[Dzious]

Hi,
Thanks to one of my co-workers, I've recently discovered the Observability tab.
Unfortunately i can't manage to make it work properly.

When i'm on the Observability page the only thing that i see is Metrics telling me that it wasn't able to fetch data.
My workflow is :
Metricbeat (ecs) -> Logstash (ecs_compatibility v1) -> Elasticsearch.
Also, i did import Metricbeats dashboards and visualisation through the config file and my system module is enable with cpu, memory, network, process, core and filesystem.

Any help would be appreciate.
Thanks by advance

EDIT : The error code is a 500 error, because kibana cannot access http://<server >:5601/api/metrics/snapshot
but i never seen anything related to snapshot anywhere

[weltenwort (Felix Stürmer)]

Hi @Dzious,

the .../snapshot API is unrelated to Elasticsearch snapshots, but used by the Metrics UI internally. Does the code 500 response contain any error message when looked at in the browser network inspector?

Another thing to check would be whether the index templates for the metricbeat-* indices were correctly installed and applied to the indices.

[Dzious]

Hi weltenwort,

Here is the error message :
message: "[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory."

Not used to web developement I did not thought about looking at the error message since it wasn't shown on my screen. Sorry about that.
As stephenb mentioned in one of my other topics, i didn't fully setup metricbeat. (forgot the metricbeat setup -e) i'll try to fix this and let you know

[Dzious]

Well even after deleting all my indexes, datastream, dashboard and visualisation and then running the command metricbeat setup -e i still have this issue.

Also i use logstash between metricbeat and elastic. I don't know if this can be part of the issue...

[Dzious]

Alright i've fixed my issue.

Here is the explanation :
Since I had workflow already setup, it was causing issue.

First of all here is my Configuration in the custom workflow i made that were causing issue :
i've changed the default name field in metricbeat configuration
I used a template to have a data stream to make thing easier for the rollover (i've read that it was a good practice, do not hesitate to tell me if i'm wrong)
I used logstash as metricbeat output

Here is how i fixed it :
So first of all. stop metricbeat and logstash.
Then disable all modules you currently have in metricbeat (just rename the file and add a .disable at the end)
edit the metricbeat configfile and set elasticsearch as output (it's just the matter of few minutes, we'll get back to logstash later on)
start metricbeat and once started use : metricbeat setup -e which will :

• create an index template
• create an ILM (i personally needed to use setup.ilm.overwrite: true because i already had one i think)
• create a kibana Index Pattern
• Create a HUGE amount of dashboard and visualisations.

Once that's done. stop metricbeat, re-enable the your modules and re-set your output to logstash.
Then go to your ILM if you already have one setup, you can delete the new metricbeat policy otherwise, you can configure it as you want.
Now configure a new index template but in the index settings and mapping you will need to copy datas from the legacy metricbeat template (you can get the json through the DevTools)

Hope this can help other people to find.
Thanks again weltenwort to guide me on the way.

[weltenwort (Felix Stürmer)]

Thanks for sharing, glad you resolved it.