Best way to get information of proccessing flow

Elastic Stack Kibana
1

Hi,

we have a logfile which holds following essential information:

timestamp, order_id, status_id

the order_id is identifying an order, which goes through different status during it's life.

Now I want to gather the information how long the order is inside which status.

What is the best way to hit this target? We are currently on following versions: Elasticsearch 1.6.0, Logstash 1.5.1, Kibana 3.1.1.

Idea 1: querying elasticsearch from logstash while processing the logmessage, requesting timestamp of previous status.

  • is that possible? How?

Idea 2: Somehow to aggregate in post processing in kibana / elasticsearch

  • how?

Idea 3: please tell me.

Thanks, Andreas

2

I suspect the best way to do this might be through the creation of an entity-centric index. This will hold a document per order, which will be updated for each new entry, e.g. through a scripted update. I think recent versions of Logstash have support for scripted updates, so doing it through Logstash would require an upgrade, but it can also be done through an external script.