Modules reported as "no data" despite data being available

[ethrbunny]

Went into "kibana" / "add data" and checked for data from a few metrics. Some appear to be working (EG system, k8) while others (EG kafka, elastic) (despite having data in dashboards, etc) show here as "no data received".

EG: elasticsearch. I go to my elastic host and check:

[root@myhost ~]# metricbeat modules list
Enabled:

elasticsearch
kibana
logstash
system

Did I miss a step somewhere?

[jsoriano (Jaime Soriano)]

Hi @ethrbunny,

Do you mean that you see data about elasticsearch, but the check for data couldn't find it? This checks just looks for data with metricset.name: elasticsearch in the metricbeat-* indexes.
Could you do this search in the discovery view to see if you can find events?
Or maybe, are you using custom index names?

[ethrbunny]

Ah ok. It's an index naming thing.

Seems like that should be flagged as needing some work. We had to break everything out into separate indexes otherwise we hit the limit v quickly.

In any event - ty

[jsoriano (Jaime Soriano)]

We had to break everything out into separate indexes otherwise we hit the limit v quickly.

What limit do you mean? What index name patterns are you using at the moment?

[ethrbunny]

The max # of items per index.

We split out most everything into it's own index. EG kafka, kibana, cassandra, etc.

[jsoriano (Jaime Soriano)]

@ethrbunny with what frequency do you hit this limit?

This is a limit per shard of about 2 billion documents, and with current defaults you should be able to store this quantity per day, as the default index template creates one index every day with one shard. These are the settings involved:

setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch.index: "metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

If you are modifying the index name check that you are still creating new indexes periodically. Other thing you can try depending in the volume of your data is to increase the number of shards.

[ethrbunny]

It isn't a document limit - it's an "indexed field" (IIRC). 1k? Something like that.

Anyway - it was easier to just break up the various inputs to their own indices.

[jsoriano (Jaime Soriano)]

Oh, yes, there is also a limit in the number of fields in the index mapping, this is 1000 by default in elasticsearch, but it is set to 10000 by default on beats indexes. This is a limit on the different fields that there can be in all the documents of an index. In principle you shouldn't be hitting this limit with metricbeat.

In any case, of course, feel free to have any indexes configuration you like