Hello
I am streaming data from my 3rdpty applications in the form of
{field1:"value",field2:"value",field3:"value"}
my 3rdpty application is a database, and the same record can change overtime with field3 being incremented but field1 and field2 remain the same (unique index on field1 and field2).
the 3rdpty will stream using logstash only the new image of the message every time field3 value is incremented.
I am trying to find a way to show in Kibana, the unique count of the pair {field1,field2} aggregating by field3. The issue i am facing is in case i have 2 messages with same {field1,field3} only consider the most recent message.
I am not sure if this is possible using Kibana, with SQL it would have been easier as i can put a condition on the timestamp in case of more than 1 message is reported.
Any idea how this can be achieved?
Thanks in advance for your help
Could you share with us what you've tried so far? Some screenshots? Are you trying to show unique count over time in a date histogram? All time unique count in a table? Samples of the data would be helpful too.
Without fully understanding your use case, my first reaction is that you might have to combine field 1 + field 2 into one outside of Kibana and do a Terms aggregation on "field3" and Unique Count on "field2".
Thanks for your reply
My goal is to create a pie chart representing the total number of the couple {package,datasetid} aggregated using count field and in case of duplicate for {package,datasetid} use the count field that is the most recent. One package can be assoicated to many datasetid and one datasetid can be associated to many packages (many to many).
The idea is to show how may dataset id have a count=1,2,3 etc.. and represent that per packages
My data follows this format:
{"package":"75d375783c8e8039f3d2191f10de7b31cf69cb4e","count":0.0,"datasetId":"hs_ppi_21052014_r96005"}
thanks!