Does elasticsearch have any query/aggregation equivalent to Splunk's streamstats search command?

Elastic Stack Elasticsearch
1

Hi,

Splunk has a very handy out-of-box streamstats search command (http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/streamstats) which calculates "rolling" statistics for each event at the time the event is seen, such as finding out "the latest MAC address" for IP address in each event, based on which it can report IP addresses whose MAC address changed over time.

Does elasticsearch have query/aggregation capabilities equivalent to that? If so, some examples will be really appreciated.

Thanks in advance,
Frank

2

I use splunk and the functionality you want is not there (at least not now) Streamstats is a python code that is processing the data that Splunk streams though it.

Now stream stats is pretty much a Term aggregation on IP with a sub-aggregation Term on MAC

Unfortuntely I don't have a Kibana console available with data to give you an example right now. But probably a data table or line chart would be what you want.

I am thinking this
Y Axis - > count
X Axis -> Chart Split Term IP
XAxis -> LineSplit -> MAC

This seems like it would produce what your asking for over what ever time period you want. Updating would just be setting your refresh rate.

3

Thanks for the reply. However as a newbie for both elasticsearch and kibana, I'm not quite following your proposal, I indexed the following events:

POST /dhcp/lease/_bulk
{ "index" : { "_index" : "dhcp", "_type" : "lease", "_id" : "1" } }
{ "time" : "2015-06-28T00:00:00", "ip" : "10.0.0.1", "mac" : "01", "lease_duration" : 10}
{ "index" : { "_index" : "dhcp", "_type" : "lease", "_id" : "2" } }
{ "time" : "2015-06-28T00:02:00", "ip" : "10.0.0.1", "mac" : "01", "lease_duration" : 20}
{ "index" : { "_index" : "dhcp", "_type" : "lease", "_id" : "3" } }
{ "time" : "2015-06-28T00:03:00", "ip" : "10.0.0.1", "mac" : "02", "lease_duration" : 30}
{ "index" : { "_index" : "dhcp", "_type" : "lease", "_id" : "4" } }
{ "time" : "2015-06-28T00:04:00", "ip" : "10.0.0.2", "mac" : "02", "lease_duration" : 20}
{ "index" : { "_index" : "dhcp", "_type" : "lease", "_id" : "5" } }
{ "time" : "2015-06-28T00:05:00", "ip" : "10.0.0.2", "mac" : "02", "lease_duration" : 50}
{ "index" : { "_index" : "dhcp", "_type" : "lease", "_id" : "6" } }
{ "time" : "2015-06-28T00:06:00", "ip" : "10.0.0.1", "mac" : "01", "lease_duration" : 60}

And creating visualization with Kibana gave me the following results:

Top 5 ip Top 5 mac Count
10.0.0.1 a1 4
10.0.0.1 a2 2
10.0.0.2 a1 2
10.0.0.2 a2 2

With splunk steamstats command I could transform:

| time | ip | mac |
| 2015-06-28T00:00:00 | 10.0.0.1 | 01 |
| 2015-06-28T00:01:00 | 10.0.0.1 | 01 |
| 2015-06-28T00:02:00 | 10.0.0.1 | 02 |
| 2015-06-28T00:03:00 | 10.0.0.2 | 02 |
| 2015-06-28T00:04:00 | 10.0.0.2 | 02 |
| 2015-06-28T00:05:00 | 10.0.0.1 | 01 |

Into:

| time | ip | mac | previous_mac |
| 2015-06-28T00:00:00 | 10.0.0.1 | 01 | null |
| 2015-06-28T00:01:00 | 10.0.0.1 | 01 | 01 |
| 2015-06-28T00:02:00 | 10.0.0.1 | 02 | 01 |
| 2015-06-28T00:03:00 | 10.0.0.2 | 02 | null |
| 2015-06-28T00:04:00 | 10.0.0.2 | 02 | 02 |
| 2015-06-28T00:05:00 | 10.0.0.1 | 01 | 02 |

Then I can filter is by "mac != previous_mac" and find out when an IP address's MAC address changed:

| time | ip | mac | previous_mac |
| 2015-06-28T00:02:00 | 10.0.0.1 | 02 | 01 |
| 2015-06-28T00:05:00 | 10.0.0.1 | 01 | 02 |

Could you please more details for me to understand if this is possible with elasticsearch/kibana?

Thanks,
Frank

4

Here is an example of me filtering on 1 ip and seeing how many Jsession id's have changed during the last 4 hours.Which is a simular idea to what you are looking to do.

Untitled.png1342×794 56 KB

y-Axis = Count
X-Axis = Split Chart (Term by True-Client-ID.raw)
X-Axis = Split Line (Date Histogram)
X-Axis = Split Line (Jsessionid.raw)

5

Hi,

Yes it's similar to my use case but not quite the same, what I'm looking for is a query that list "all" IP addresses whose MAC address got changed over time. Thanks for the posts anyway and I'll keep exploring possible solutions.

Regards,
Frank

6

Yah if you want to do comparisons of different records then at Kibana's level you can not do it. Kibana can only show you aggregations of data but not 2 different documents. Oh but you can check this Logstash Filter Project, it is new but might help you get all your information in to 1 document.