I found this message in one of the HTTP requests when loading the metrics page.
[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [host.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
But host.name in my case is already a keyword field so I don't know what to do next. Does anyone know how to solve this problem?
is it possible that there are indices matching filebeat-* in which host.name is not a keyword? The Infra UI tries to access the configured log indices for the "log rate" metric, so there might be a mapping conflict between the metricbeat and filebeat indices.
Does filebeat-* affect the metric page? I was under the impression that only the metricbeat-* affects the page. I tried to set it so it only looks for a specific index such as metricbeat-7.1.1-2019.05.30-000001.
There are some queries where it accesses the log indices too, which is why they share the same source configuration. Could you try out one of the following:
If you have filebeat-* indices and use the Logs UI, make sure these indices also have the correct ECS-compatible mapping.
If you have filebeat-* indices and don't use the Logs UI, set the log indices source configuration to something that doesn't exist, e.g. no-index-with-this-name-*.
If you don't have filebeat-* indices, we have to look for a different cause.
I just deleted the filebeat-* indices which weren't used anymore and it seems like the page still displays "There is no data to display". When I check the browser logs, I found these:
Content Security Policy: Directive ‘child-src’ has been deprecated. Please use directive ‘worker-src’ to control workers, or directive ‘frame-src’ to control frames respectively.
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). infra:1:1
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). infra:372:1
^ A single error about an inline script not firing due to content security policy is expected! bootstrap.js:10:20
window.controllers/Controllers is deprecated. Do not use it for UA detection. vendors.bundle.dll.js:62:739542
As mentioned, a single error is expected. But in your case there seem to be additional problems with how the browser handels CSP. What version of which browser are you using?
Those errors are probably not related to the problem of data not showing up. The error message about the fielddata being disabled persists even though you deleted the filebeat indices? What are the "metric indices" and "log indices" settings you are using?
I noticed the host.name on my logstash index is of type text that also has the field keyword. But if I set the Host name to host.name.keyword on the infrastructure page, I can't see my hosts and get into the metrics page.
Looks like not having host.name being of the keyword type in the logstash index is indeed the cause. The fact that you have a host.name.keyword field indicates that you are using a dynamic mapping, because that is what Elasticsearch infers for text fields by default.
Are you using logstash to forward log entries read by filebeat or do you perform custom log parsing with logstash?
Yes, I use Logstash to remove color codes from logs so the raw color code characters don't appear in the Kibana's log page.
Do you have any recommendation on what to do to satisfy the metric page? I find it odd that it needs the log data (e.g. what would happen if I don't send logs to Elasticsearch and just the metricbeat data?).
If the there were no indices that match the "log indices" settings, there shouldn't be any influence on the metrics.
I would recommend to use an appropriate static mapping for the log indices too. When sending the log entries from filebeat to Elasticsearch via logstash, though, filebeat can't install its index template. I would recommend to run filebeat setup --template to install the index templates and to preserve the filebeat-* index names even when sending via logstash.
If you only use logstash for performing the message processing as you described, you might want to look at performing this in an gsub processor in an ingest pipeline in Elasticsearch directly. That way you could avoid having to run a separate logstash instance and would reduce the risk of running into mapping problems.
Do you mean the metrics should show if I use non-existent index? If I set log indices to look for non-existent index, it shows this error on the metrics page:
I could reproduce the effect and it looks like a bug. I will file a GitHub issue. In the meantime, try using a wildcard in the logs index setting, e.g. qwerty*.
Thank you for patiently walking me through the process of reproducing it. I have created a fix in https://github.com/elastic/kibana/pull/38976. Feel free to verify it or just follow the PR progress.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
