Error in index or visualisations for "[Metricbeat System] Overview ECS"

[bodo.te (Bodo Teichmann)]

After setting up ELK 7.3.1 according to ELK Tutorial :
https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html
and looking in to this dashboard:
"[Metricbeat System] Overview ECS", I got several errors in the dashboard , e.g.: wrong indication of "Number of hosts [Metricbeat System] ECS" = 0 or other errors, see screenshot below. . Looking into elasticsearch logfile I see the message:

 " org.elasticsearch.transport.RemoteTransportException: [8c0c66034037][172.22.0.5:9300][indices:data/read/search[phase/query]]
 Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [host.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

when i try to fix this using this by setting "fielddata=true" according to this description: https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html
i.e. :

PUT metricbeat*/_mapping
{
  "properties": {
    "host.name": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

all i get is another error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
  },
  "status": 400
}

now I'm lost.

The only workaround I found , was to manually change lots of fieldnames in all visualizations of the dashboard:
e.g. change in visualisation "Number of hosts" the fieldname from "host.name" to "host.name.keyword"
this is however very tedious. Since there are about 10 shards to change, everytime I setup a new ELK-Stack.

[stephenb (Stephen Brown)]

Hi @bodo.te
Sorry you are having trouble let see if we can get this straightened out. Once we get this straight you only need to setup metricbeat once and you will not need to do the "editing / fixing" you are trying to do.

To me it looks like you loaded the dashboards but not the index template and index pattern which is required to set up the correct mappings (which I suspect is the source of your issues)

The directions have set them up separately but I suggest just using the following command to setup everything at the same time

./metricbeat setup

the above command loads the dashboards index templates index patterns all in one command.

https://www.elastic.co/guide/en/beats/metricbeat/current/command-line-options.html#setup-command

However I suspect that you are in a half set up state at this point and the error about the field data is means the mapping and index patterns are not correct.

Unfortunately to get back in a good state we will need to delete the metricbeat indexes, metricbeat index template, and metricbeat index pattern and then start over by running the command above.

I am assuming this is just test data so that you can delete it....

So here is what we are going to do...

1. stop all metricbeats on all hosts

2. Navigate to Kibana / Management / Index Patterns
   And delete the metricbeat index pattern by selecting it then deleting it.

3. Navigate to Kibana / Dev Tools
   Run the Following 2 commands, it is possible the command will fail because the template does not exist

WARNING This is going to delete the existing metricbeat data do NOT do if your need this data.

DELETE /metricbeat-*

DELETE /_template/metricbeat-7.3.1

4. Go to your metricbeat host and run

./metricbeat setup

then

./metricbeat -e

5. Navigate to the Visualizations, you may need to force refresh / reload Kibana but you should see the correct visualizations.

Note the easiest way to get metricbeat or any beat running is just to follow the quickstart guides that are built into Kibana

Example:

http://localhost:5601/app/kibana#/home/tutorial/systemMetrics?_g=()

http://localhost:5601/app/kibana#/home/tutorial_directory?_g=()

Hope this helps

[bodo.te (Bodo Teichmann)]

Yes, thank you! Following your advice: everything is ok now.

But I still don't get it, whats going on.
I guess the only thing, I did differently is to start the metricbeat first (using systemd's start ) and then , after a while, I executed the "metricbeat setup" command. Why does this lead to the issue I had before?

[stephenb (Stephen Brown)]

I guess the only thing, I did differently is to start the metricbeat first (using systemd's start ) and then , after a while, I executed the "metricbeat setup" command. Why does this lead to the issue I had before?

Yup that would be the problem... for any beat you must run setup first (just once not on every host) .. it sets up many things... including the template which defines the mapping (which you can think of the schema).

When you just run metricbeat or anybeat without the setup Elasticsearch will index the data anyways but not with the exact data types etc that are needed to use all the out of the box visualization, dashboards, index lifecycle management etc.

Glad you got it working!

If you add any more data just follow the instructions that can be found at

http://localhost:5601/app/kibana#/home/tutorial_directory?_g=()