Timelion with live streaming

[areyja]

Hello ELK Gurus,
Pl excuse my ignorance if this is very common issue and mistakes I am doing. I am 1 week into ELK.

I have set ELK on a single RHEL server. I have jdbc input from logstash sending data to elastic. I took care of doc_id so I do not have dups pumped. I am trying to use this live data in timelion. However due to nature of my logstash(jdbc input) schedule (* * * * *), logstash is in collecting, dying, resurrecting, collecting loop.

As a result, if you see the right side of time graph, the count is very high. shown by encircled area. As time passes by, or when window moves, the data normalizes fine, however at the edge of the graph it is messed up. It always remains high like shown in pic

May be it is messed up because of time(2-3 min) elastic is taking to resolve/normalize incoming stream? What should I do handle this?

Regards,
/a

[Marius_Dragomir]

Timelion is basically displaying what it's coming from elasticsearch. The best thing to do would be to offset the data slightly so that it only displays the normalized data.

[areyja]

Thank You. Yes make sense. In my case offsetting it wasn't an option.
I had to track column to not read what was read before. Relieving elastic ingesting and normalizing a large volume almost every minute.

/a