I seem to be running into limitations on Elastic and Kibana

Elastic Stack Kibana
1

I am hoping that someone can set me straight. I have Googled, and watched videos, to my heart's content. I need to know how to do more advanced analysis of elastic documents. For example,

Let say we have logs that look like this:

user1 "Successfully Logged in"
user2 "Successfully Logged in"
user3 "Successfully Logged in"
user1 "Successfully Logged out"
user4 "Successfully Logged in"
user2 "Successfully Logged out"
user5 "Successfully Logged in"
user6 "Successfully Logged in"
.
.
.
user50000 "Successfully Logged...."

You get the point, I need to know how many users are still logged into a system. So I need to find the last update for each user, and then count how many of those are "logged in". That requires multiple passes over data, and then either some math, or a simple count of the results. I cannot find any evidence that elastic can handle this.

Then there is another problem, let's call it problem number 2.

Let's say you have that same log from above. But I need a dashboard that shows how many level 1 sales reps are logged in, and how many are from the state of Texas. That info is stored in Active Directory so I need to either look that up at the time I build a dashboard OR at the time I store the document. I see nothing that says I can even do it at DB time, and very little information about . how to do that at document creation time that does not have severe impact on performance when you are talking about 60,000 users pushing thousands of transactions per second. I was hoping for a lookup table that could be updated once a day that could be accessed quickly for inbound records.

Are either of these things possible with Elastic & Kibana? I know our company spends a great deal of money on this technology and I have been a big supporter internally, but I am running into these situations more and more and need to find better sources of information on how to solve them.

Thanks for your input!

2

I should add that I already read over the translate filter of logstash. And it seems to fit exactly for problem #2. BUT as mentioned I just need more info, or someone that is an expert to explain how I would go about using one field to add two additional fields.. ie.

user1 would add
location : texas
title : super fantastic salesperson, level 1000

I should have mentioned that I already read over that.

Thanks again.

3

Elasticsearch and Kibana generally expect you to structure your documents to contain all the relevant information in a single document, denormalized as much as possible. The idea you're talking about is sometimes referred to as "entity-centric modeling", and the only solution we really offer in that area is data transformations: https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html

These transformations are intended to solve cases like web requests or user sessions, so you should check that out.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.