Negative Filter - Not getting expected results

[Elastastic (Mike Reed)]

Hello all,

You'll have to excuse me as I'm quite new to the ELK stack and so, I might not be using proper terminology when describing my problem so please forgive me in advance.

I'm running a new installation of ELK on an Ubuntu 16.04 (Server build).

Version numbers are:
elasticsearch - 6.2.3
logstash - 1:6.2.3-1
kibana - 6.2.3

We're using this stack to attempt to analyze successful/failed user logins within our Windows domain. Along with this new ELK install, we've got a Windows domain controller (2012 R2) successfully sending login events via winlogbeat (6.3.2) to the stack through elastic (I've got logstash configured but in the interest of time, I wanted to just get things into elastic without trying to over complicate things via logstash).

To make a long story short, I'm targeting the "event_data.TargetUserID" field within each event to build a quick graph of the top 10 successful/failed user accounts (these are being pulled from Windows event_id 4624).

The issue I have is that the results from my query return mostly computer accounts in the form of "COMPUTERNAME$". Because we have so many of these computer accounts, they are always the top 10 or 15 results and thus, I'm not getting any meaningful user account results.

So, I tried to add a negative filter which I thought would return only accounts that don't match "COMPUTERNAME$" and I can't seem to find the proper syntax. I understand that I can use regex queries and I know that lucene regex matching has a few differences but no matter what I try, I can't seem to filter out these computer accounts using one or two filters.

Some examples of accounts I don't want to see are (most of the time entries are uppercase):
COMPUTERNAME$
COMPUTER-NAME-EXAMPLE$
COMPUTER-NAME$

I thought a simple regex such as ".*$" would work but no luck.

I've also tried:
.$
.$$
(?i).$
(?i).$$

When running these through a regex checker, they seem to match what I want but when running the negative filter in Kibana, I still get many results in the NAME$ form.

Does anybody know how I would construct a filter to remove anything in the form of "NAME$/COMPUTER-NAME$/COMPUTER-NAME-TEST$" from my query? Basically anything with $ at the end of the username.

When I add a negative filter and specify a specific hostname (SYSTEM$), I do get proper results which do not show that hostname but when I try any type of negative wildcard filter, I still get back many of the results ending with "$" that I don't want.

My apologies for the long post but I wanted to give as much info as possible.

Thank you in advance for the help and support.

Cheers,

Mike

[ppisljar (Peter Pisljar)]

Hey,

the question has many answers, ill try to provide two to get you started:

• using lucene syntax https://www.elastic.co/guide/en/elasticsearch/reference/6.3/query-dsl-query-string-query.html#query-string-syntax
  you could write your query something like:
  -event_data.TargetUserID:COMPUTER*
  note the - infront (to indicate to return everything but documents matching the query)
  note the fieldname:fieldvalue ... collon, and no spaces
  regex are tricky .... wildcard searches as well

now to the second part of my answer .... you tried to search your data by the inital wildcard ... like everything ending with $ ... if you think for a while how this search will work:

• it will go thru EVERY record in the dataset and check if the last character is $ .... this doens't scale. checking every record in the database will not be fast at all.

• this would however work if you would index your words in reverse .... then you are searching by the first character again. elasticsearch has a reverse token filter to help you with that https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-reverse-tokenfilter.html

for more information on this i suggest you to pop to elasticsearch discuss forum.

[Elastastic (Mike Reed)]

Hello Peter,

Thank you for your reply and my apologies for the delayed response.

As I've gotten further into this (and after a fair amount of reading), I suspect the regex query is what I'm really looking for. I tried many various searches using wildcard characters but wasn't able to achieve what I wanted.

After trying some regex queries, I'm a bit closer but still am getting COMPUTER$ in my results, regardless of what type of search I run (wildcard/regex).

I'm wondering if the query results have anything to do with how the events are being analyzed and more specifically, how the tokenizer is formatting the value of the fields after the event has been ingested.

Thank you as well for the reverse token suggestion, I'm looking into that now.

For now, I'm not sure what exact question to ask to help solve my problem so I'll go ahead and close this one.

Thank you again for your help and support.

Cheers,

Mike