I was previously using elapsed filter in my logstash config to calculate time difference of start and end event. Recently I have replaced elapsed filter's code with elasticsearch output filter and painless script to do the same calculation.
However when our infrastructure team has deployed the new logstash config, we're seeing some events on Kibana have old elapsed_time field and its associated tags and some events are showing new fields that I have recently introduced!
The config was deployed on two logstash hosts and I checked thoroughly and couldn't find any of the configs including mine that has reference to elapsed filter.
Data flow is like this:
filebeat -> haproxy VIP -> 4 haproxy -> 2 shipper -> 4 kafka -> 2 logstash instances where my code is configured -> elasticsearch
Our ELK and logstash version is 7.6.2
Logstash service was restarted but we haven't rebooted the server.
Are you aware of any bug that allows logstash to run old code from cache? How to resolve this issue?