Compare incoming string/data to keyword list, if in keyword list, flag?

Elastic Stack Logstash
1

hi,
i'm not really sure the best way to describe what i'm trying to do.. but i will have data bring streamed into logstash. i have a growing keyword list, that is a text file.

is there a way i can compare the values/data coming in to this keyword list. if the data coming in contains values from the keyword list it creates a new boolean field? keyword=true?

does that make sense?

any suggestions would be appreciated

2

Do you have control of the input keyword file? (Can you modify the format?)

If so, you could probably make use of the filter translate.

If each line of the dictionary is a regex like. '.*mykeyword.*', and the value associated is 'true', it could probably work (even if I'm not sure it would be the best in term of performances).

3

i've looked at that approach, and the problem is, the keywords will have match WITHIN the datastream.

example: keyword: "blogspot.com"
androidlover.blogspot.com MATCH
iphonelover.blogspot.com MATCH
nokia.blogspots.com
samsung.blogspot.com MATCH

4

That's why I talked about dictionnary using regex keys :smile:

Using your example (and assuming we read the dic value from a file), we would have something like :

filter {
  translate {
    field => "[message]"
    destination => "[match]"

    dictionary => {
      "blogspot\.com" => true
    }

    fallback => false
    regex => true
  }

}

Become :

{
  "@timestamp": "2019-12-19T18:43:57.977Z",
  "@version": "1",
  "host": "localhost",
  "match": "true",
  "message": "androidlover.blogspot.com"
}
{
  "@timestamp": "2019-12-19T18:43:57.978Z",
  "@version": "1",
  "host": "localhost",
  "match": "true",
  "message": "iphonelover.blogspot.com"
}
{
  "@timestamp": "2019-12-19T18:43:57.978Z",
  "@version": "1",
  "host": "localhost",
  "match": "false",
  "message": "nokia.blogspots.com"
}
{
  "@timestamp": "2019-12-19T18:43:57.978Z",
  "@version": "1",
  "host": "localhost",
  "match": "true",
  "message": "samsung.blogspot.com"
}

Did I understand you correctly ?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.