Dashboards and visualizations seem out of date

[CW360_Michael (Michael)]

After upgrading everything to 6.3 the only issues I'm seeing are with Filebeat dashboards and visualizations.

Here's a screenshot showing the UI throwing fits:

I've removed and reinstalled all dashboards, visualizations and indexes by manually clearing things in Kibana and re-running filebeat setup. I've also refreshed field lists.

When I edit the dashboards and visualizations and change something like "nginx.error.level" to "nginx.error.level.keyword" things work as expected. Manually updating visualizations and dashboards in most cases seems to fix things but I was hoping they'd have already been updated as part of 6.3.

Am I doing something wrong? Is there a proper way to get updated dashboards/visualizations for 6.3?

From going through the forums I know that log output for ES is asked for and not often provided. I'm providing it at https://gist.github.com/ActionScripted/abe19281756df0a73ca818405f003e37 for good measure, but this seems to me to be an issue of outdated dashboards/visualizations leading to ES errors.

[CW360_Michael (Michael)]

Using https://github.com/elastic/beats/blob/master/filebeat/module/nginx/_meta/kibana/6/dashboard/Filebeat-nginx-overview.json I've been manually updating things to slowly get everything working again.

Any time I see something like "terms_field": "nginx.access.url" in the visualization JSON I replace it in Kibana with nginx.access.url.keyword and it starts working.

[CW360_Michael (Michael)]

Was able to get everything but the access map and data volume visualizations working.

Not sure how to edit the JSON to use the split lat/long for the map (old version used nginx.access.geoip.location) and am not able to use nginx.access.body_sent.bytes as a sum-able item.

Still wondering if maybe I am or have done something wrong.

[pierhugues (Pier-Hugues Pellerin)]

I've tested with the latest 6.3.0 release of the stack and I don't run into this issue and looking at your ES error it looks like an error with fielddata, but we don't enable it for any nginx fields.

Can you gist your mapping? http://localhost:9200/filebeat*/_mapping for this index?

[CW360_Michael (Michael)]

@pierhugues thanks for the quick follow-up!

Here's the Gist obtained from Dev Tools > Console > GET _filebeat*/_mapping: https://gist.github.com/ActionScripted/93c8bfd299006af92135e0630f55fee9

(When we moved to 6.3, we cleared old 6.2.4 indices.)

[CW360_Michael (Michael)]

After getting annoyed with the constant errors, wiped everything out and started fresh. Removed all Kibana dashboards and visualizations along with all Filebeat and Kibana indices (including config ones).

Once all Kibana and Filebeat-related data had been purged from ES, changed Filebeat config to use ES instead of Logstash and re-ran Filebeat setup (filebeat setup).

Not sure what went wrong where. Maybe we did the upgrade out of order or something. Now that we completely wiped Kibana and Filebeat and re-setup Filebeat things are working as expected without issue.