Filter out signals that are produced too often by one user

Elastic Stack Kibana
1

Hi all,

I am working for a software company and we use Elasticsearch and Kibana to monitor the signals that our clients send. For that we implemented signals, that include a UserID and unique errorcodes.

The problem is, that there are some few clients, that send a lot of the same errorcodes when they run into a loop (this happens frequently, like once every two weeks one user produces thousends of signals). So now, looking at the number of events (COUNT) it seems like we have an error even if the functionality of the product is not really broken, but only one user producing a large amount of errors.

So my question is: is there a possibility to automatically filter those few users out and exclude them from my visualisations ? Like creating a flag for this users that I can easily exclude?

At the moment the only possibility I can see is to filter for those users manually, but this would take a lot of time and as we always get new data I would have to check this every single day.

We also think about doing such implementation server-side, but it would be much easier if we could use a Kibana-side implementation.

I Would be thankful if you have some ideas and could help me.

Best regards,
lina

2

You could run an ML job to highlight those anomalous users. But then there's currently no way to automatically connect that find back into other areas at this stage (or none I can think of that is).

You could setup a generic filter for counts < $somelargenumber that'll just ignore those?

3

Thank you for your reply!

I also thought on setting filters, but as far as I know, the only possiblity to filter would be on one field. But the filter I would need is Count(documents grouped by one user and one errorcode) < $large number. If I only filter out high count for all users and all codes thogether I could miss if something goes completely wrong with the software, right?

4

Yes, that's correct.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.