I was experimenting with Elasticsearch/Kibana to replace our existing Prometheus/InfluxDB/Grafana setup for monitoring Kubernetes and collecting system logs.
I used the Kubernetes Operator to deploy an instance, pretty much following the official tutorial.
This all worked pretty well, except for some trial and error in how to actually configure Beats deployed using the operator!
After clicking around in Kibana, I noticed that it came with a bunch of pre-configured Dashboards, like Metricbeat-system-overview-ecs. These did not always work for my setup, however:
- I deployed Metricbeats twice, once as a DaemonSet and once as a Deployment with a single replica, which caused the pod of that replica to show up as a Host
- I set the period on metricbeats to 30s. This caused all kinds of visualizations to break, since it defaulted to 10s buckets.
- I liked to graph the Derivative of some stats more than the total, so I changed the Aggregation in some places
- There is this Navigation Overview Panel on top of some Visualizations, which I edited to have more links.
In short, I took the pre-installed Dashboards and changed them a lot. I made sure to click on Save both for the Visualization and the Dashboard every time. I edited the Visualizations from within the Dashboard edit mode.
Today however, all my changes where gone!
This took a long time to setup (I spent almost the entire day on it), so before doing all of it again, I wanted to first ask if ES/Kibana is even capable of persisting anything for longer than a browser session.
Here are some observations I made:
- I'm not careful about my browser usage. I opened new tabs whenever I liked and used the browsers back functionality a lot.
- Both servers never restarted. There were some exceptions logged, though.
- I run a single replica of elasticsearch.
- Everything I copied instead of editing is still here. Since I took a while to figure out that Visualizations are their own Entity and are shared between Dashboards, I almost exclusively copied the Dashboards, which now contain a bunch of broken visualizations.
Has anyone experienced this before? Are the built-in things getting "reset" or something? All my logs/metric data is still here, and things I cloned persisted, so I highly doubt my there is a general problem here, dispite my earlier comment
Thanks in advance!