Kibana loses visualization edits?

[jreusch]

Hi folks!

I was experimenting with Elasticsearch/Kibana to replace our existing Prometheus/InfluxDB/Grafana setup for monitoring Kubernetes and collecting system logs.

I used the Kubernetes Operator to deploy an instance, pretty much following the official tutorial.

This all worked pretty well, except for some trial and error in how to actually configure Beats deployed using the operator!

After clicking around in Kibana, I noticed that it came with a bunch of pre-configured Dashboards, like Metricbeat-system-overview-ecs. These did not always work for my setup, however:

• I deployed Metricbeats twice, once as a DaemonSet and once as a Deployment with a single replica, which caused the pod of that replica to show up as a Host
• I set the period on metricbeats to 30s. This caused all kinds of visualizations to break, since it defaulted to 10s buckets.
• I liked to graph the Derivative of some stats more than the total, so I changed the Aggregation in some places
• There is this Navigation Overview Panel on top of some Visualizations, which I edited to have more links.

In short, I took the pre-installed Dashboards and changed them a lot. I made sure to click on Save both for the Visualization and the Dashboard every time. I edited the Visualizations from within the Dashboard edit mode.

Today however, all my changes where gone!

This took a long time to setup (I spent almost the entire day on it), so before doing all of it again, I wanted to first ask if ES/Kibana is even capable of persisting anything for longer than a browser session.

Here are some observations I made:

• I'm not careful about my browser usage. I opened new tabs whenever I liked and used the browsers back functionality a lot.
• Both servers never restarted. There were some exceptions logged, though.
• I run a single replica of elasticsearch.
• Everything I copied instead of editing is still here. Since I took a while to figure out that Visualizations are their own Entity and are shared between Dashboards, I almost exclusively copied the Dashboards, which now contain a bunch of broken visualizations.

Has anyone experienced this before? Are the built-in things getting "reset" or something? All my logs/metric data is still here, and things I cloned persisted, so I highly doubt my there is a general problem here, dispite my earlier comment

Thanks in advance!

[timroes]

Hi,

what you are describing sounds def like some sort of bug. Kibana is saving it's visualizations and dashboards inside the .kibana index in your Elasticsearch cluster, whenever you hit save. Also we keep a temporary state around via the URL, so you can actually navigate forth and back in Kibana without losing your state (in most applications) before actually saving them.

Kibana also has no problems working in as many tabs in parallel as you want. I mean you could potentially overwrite some changes if you open the same visualization in two tabs making different changes and than saving in both tabs (the later change will just overwrite the first one), but it doesn't sound like this would have been an edge case you ran into.

Did you potentially inject the beats dashboards (via the --setup command) again after making your edits (which could potentially overwrite some of them)? Maybe there is some persmission errors going on. Since you mentioned some exceptions were logged it might be helpful if you could copy them, especially also if the Elasticsearch output produced any errors.

In general there should be no reset going on, nor should Kibana lose your visualizations, Dashboards. As a general recommendation (to not potentially clash with further setup of beat modules), it might help to clone the dashboard completely before making edits to it.

Cheers,
Tim

[jreusch]

Maybe I can use Kibana to figure out what's wrong with it!

This shows the updated_at column of the visualizations in .kibana_1:

And here are requests logged by Kibana containing "visualization":

All the updated_at timestamps are exactly at the time I took a break! There is noone else with access to Kibana currently.

Additionally, I found a lot of messages like these in the log:

{"type":"log","@timestamp":"2020-10-08T18:50:52Z","tags":["warning","plugins","visTypeTimeseries","visTypeTimeseries"],"pid":7,"message":"Request validation error: [panels.0.series.0.metrics.0.size]: types that failed validation:\n- [panels.0.series.0.metrics.0.size.0]: expected value of type [string] but got [number]\n- [panels.0.series.0.metrics.0.size.1]: expected value to equal [null] (saved object id: unsaved). This most likely means your TSVB visualization contains outdated configuration. You can report this problem under Sign in to GitHub · GitHub"}

The Elasticsearch logs don't contain any entries for this time frame, the exception I mentioned earlier was a cancelled search at some other time, apparently.

I don't know what parameters I pass, since its managed by the Operator. I used this to setup Kibana and Elasticsearch, and according to Kubernetes, it never restarted since:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 7.9.2
  count: 1
  elasticsearchRef:
    name: logs
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logs
spec:
  version: 7.9.2
  nodeSets:
    - name: default
      count: 1

      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false

      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data

          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi

It still kind of looks like it reset all the default dashboards while I was on break. I definitely didn't manage to update 10 Visualizations per second, If I could, I would have just done it again.

Today, all my changes are still here, so I think I will try to create all Dashboards manually in a different Space, and see if it happens again.