Historical trend chart (project related)

[delly.fofie]

Hello!
We are using Kibana to display the evolution of the projects. A project in our case is a set of requirements with different States for e.g (In preparation, Approved, Implemented).
This image show an example:

The problem is, on 20.07.2018, we are storing the objects with their corresponding states, so total object is 7
But on 21.07.2018, the first object's(o1) state changed from implemented to approved.
And whe have liked Kibana/Timelion to display in area trend charts the correct counts for each day for the three States.

I see here two possibilities:

1. Re-importing the data every day (Not only the one which change) -> Data grow to fast.
2. Maintaining a second index which store a count for each given state on a daily based -> User can only see the count and not the objects).

Is there another elegant way to solve that? Any Idea ?
Thanks in advance.

[NerdSec (Nachiket)]

Hi delly,

Not sure that i completely understand the issue, but I will try.

You can import the data every day using something like Logstash easily. Creating a new record every day should easily allow you to create a trend chart. Is there any issue in doing so?

[delly.fofie]

@NerdSec

You can import the data every day using something like Logstash easily. Creati

Which records are you saving on the second day ? Not the one which changed ? Or the whole set ?
Thake a look again at the image I attached to the post, how can you let elasticsearch know that on 21.07 it remains:
2 objects with the implemented state ?

[NerdSec (Nachiket)]

Hi,

You should index all records each day. Elasticsearch indices are usually time-based, so the timestamp would be the time of ingestion in elasticsearch.

Now, when you create a visualization in Kibana, you can select the timestamp to something like "Current Day". This will allow you to see the status of your projects as on date. For trends, you can select a longer time period, thereby enabling you to see the older data.

Hope this helps.

[delly.fofie]

@NerdSec

e

thanks for your reply. It's not a how Elasticsearch/Kibana works questions. But mainly on how to structure the data in order to get the trend chart i want. What you are explaining here is mostly how to set up kibana/ES

[NerdSec (Nachiket)]

correct!

You could also assign a unique id to each project and then consume it daily, but then all data will be over-written and you would not be able to show the trends.