Metricbeat - mapping conflict

[anhlon]

Hi All,

I am running into a problem with metricbeat. In my container I have a label "org.label-schema.build-date" which is set to the container build date. Things appear to be working fine, except over time, it causes a mapping conflict.

First, the Index Pattern will show these two fields "docker.container.labels.org.label-schema.build-date" and "docker.container.labels.org_label-schema_build-date" the difference here is the period and the understore.

Secondly, on certain daily index, one of them will be showing the type as "date" while other days will show it as "keyword"

When this happens, my dashboard will fail to show any data. What is the recommended approach to fix this? ie, make it one type or the other and not both?

Thanks

[jsoriano (Jaime Soriano)]

Hi @anhlon,

In principle dots in labels collected with the metricbeat docker module are replaced by underscores, and the values are stored as keywords. The labels.dedot setting, enabled by default, allows to disable the replacement of dots by undersocers.

If you also enable the add_docker_metadata processor, it can additionally try to set the fields, would it be possible that you are using both, the docker module and the add_docker_metadata processor?

[anhlon]

ah ha!

I am using metricbeat.autodiscover and attaching add_docker_metadata in the processors section, ie

metricbeat.autodiscover:
  processors:
    - add_docker_metadata:  

now that you have pointed out my silly mistake, I'll go and remove it.

thanks!

[jsoriano (Jaime Soriano)]

This is not a silly mistake
add_docker_metadata also has its use cases enriching events of non-docker modules, it is only when used on docker events when it can cause conflicts, as both features collect the labels. We will try to make it less prone to mapping conflicts in the future.

[anhlon]

Thanks Jaime,

Another question if you don't mind around MetricBeat and it's dashboard. Once I removed the add_docker_metadata I no longer get the mapping conflict. However, I am seeing 2 version of a same field, in different variation. For example docker.container.labels.com.amazonaws.ecs.container-name and docker.container.labels.com_amazonaws_ecs_container-name. Which one should I be using if I were to create dashboards and using that to split the data?

I am putting the data in a daily index, a dashboard in kibana works for one day, the following it will fail with "Courier fetch error", using developer tools, I can see errors saying that
"fielddata must be set to true on rabbitmq.node.name" it then suggests that I use keyword.

The problem that I am seeing is that I can properly view the dashboard on day one, the next, it will not show anything and give me a "Courier fetch error". All the dashboards are canned dashboard from Metricbeat. Would this contribute to the error?

output:
  elasticsearch:
    hosts: ["localhost:9200"]
    index: "metricbeat-${ENVIRONMENT}-6.3.0-%{+yyyy.MM.dd}"

setup:
 template:
   enabled: true
   name: "metricbeat"
   pattern: "metricbeat-${ENVIRONMENT}-*"
   overwrite: true

Now I have different environment, say development, test, qa . would having the same template name causing an issue? Should the template name be something like "metricbeat-${ENVIRONMENT}"?

Thanks

[jsoriano (Jaime Soriano)]

@anhlon

Another question if you don't mind around MetricBeat and it's dashboard. Once I removed the add_docker_metadata I no longer get the mapping conflict. However, I am seeing 2 version of a same field, in different variation. For example docker.container.labels.com.amazonaws.ecs.container-name and docker.container.labels.com_amazonaws_ecs_container-name . Which one should I be using if I were to create dashboards and using that to split the data?

If you are only using the docker module, it "dedotes" by default, so the fields should be named after the label, but replacing the dots by underscores, i.e. docker.container.labels.com_amazonaws_ecs_container-name in this case. You can still be seeing the labels with dots if you had a metricbeat writing to the same index (or to an index that matches the same index patter in Kibana) with the add_docker_metadata processor enabled, or if you set dedot: false in the docker module of any of your metricbeats.

@anhlon

I am putting the data in a daily index, a dashboard in kibana works for one day, the following it will fail with "Courier fetch error", using developer tools, I can see errors saying that
"fielddata must be set to true on rabbitmq.node.name" it then suggests that I use keyword.

Are you seeing this error in a dashboard provided by metricbeat? What query are you running in the developer tools to see these errors?
Problems with fielddata can indicate problems in queries and/or in field mappings, see more about this here.

@anhlon

The problem that I am seeing is that I can properly view the dashboard on day one, the next, it will not show anything and give me a "Courier fetch error". All the dashboards are canned dashboard from Metricbeat. Would this contribute to the error?

output:
  elasticsearch:
    hosts: ["localhost:9200"]
    index: "metricbeat-${ENVIRONMENT}-6.3.0-%{+yyyy.MM.dd}"

setup:
 template:
   enabled: true
   name: "metricbeat"
   pattern: "metricbeat-${ENVIRONMENT}-*"
   overwrite: true

Now I have different environment, say development , test , qa . would having the same template name causing an issue? Should the template name be something like "metricbeat-${ENVIRONMENT}"?

I don't think this can contribute to the error as the template is probably going to be the same if you are using the same versions for all environments, but it can be a good idea to use a different name for each one of them, just in case. For the metricbeat version, you can use %{[beat.version]} instead of harcoding it to 6.3.0.

[anhlon]

Thanks for the info.

I figured out all my problems! I ended up changing
pattern: "metricbeat-${ENVIRONMENT}-"
to
pattern: "metricbeat-"

and the dashboard problem goes away. and I had no idea why i wasn't using "%{[beat.version]}"