We would like to create more insight into the processing of orders in our systems. The incoming orders go through several steps (processes). We would like to monitor all these steps using ELK. The goal is to create insights about the status of the orders: Have they been completed? If not, which step did it fail to complete?
As of yet we are not interested in troubleshooting errors. We only want to know which of the processes was the latest to "run".
- The order is received by the system. It has a unique orderID
- The order is processed in system 1
- The order is processed in system 2
- The order is sent out and marked as "processed"
For all 4 steps we managed to import the transaction data. Now we want to make the correct connection and analyze the data. We were thinking of creating a datatable such as this:
OrderID; Received; system1; system2; completed;
1234; Yes; Yes; Yes; Yes;
5678; No; Yes;
In this example, we would see that order 5678 was not completed. Currently we don't know how to create such a data table yet. Best we can do with the data table vizualisation is this:
Order ID; Filter_ServiceName;
This datatable would not be userfriendly, as it does not offer quick insight into which services are not processed (you can't sort on empty values for the "completed" process). Is there any way to create the first table?
As an alternative, it would also be good if we could create a query, where we could search for all orderID's that are present in the first step ("received"), but not present in the last step are "completed" yes. Is it possible to make such a query?
Note: every step in the process flow is written into ES as a seperate transaction. For each complete transaction, we will have 4 transactions in ES.
I would love to hear your ideas in how to solve this problem.