I am building a TSVB visualization. Everything works great up until the point where if the application is restarted/restarts on its own the counters are reset to '0'. Since the visualizations use derivate to calculate difference in metrics, after this reboot, I get a huge negative spike in graphs which makes the rest of the metrics useless. (more details here on the use of derivative: Time Filter behaving strangely)
So my plan of action was going to be drop the first event after counters reset to avoid this negative spike but i dont think it will work because the metrics would take time to get back to last value before restart. For instance: Last value of SUCCESS queries was 55,000 before counters reset and until the value becomes higher than 55,000 the graph would always be negative due to derivative calculating difference from last event.
How should I tackle this? I am open to changing visualizations (5) completely if that would solve this.
Additional information about the data:
I am pulling in json data every 10 minutes using http poller input plugin. In the data, there are 2 fields: boot-time and current-time. Current Time refers to poll time and Boot Time is when the application was turned on.
The metrics are maintained since boot time and always increment up. So there is no good way of showing a trend unless I use derivative.
Also, is there something that can be done at logstash level that can calculate diff between the metric and pass that to ES?