Read list within logstash configuration (translate plugin)

Elastic Stack Logstash
1

Hi guys,

I have been working with the ELK stack with great succes for a project! really thanks for the great stuff!

Recently, I would like to try have a go with using logstash to read a list of things to compare to, but I am stuck.

The situation is as follows:

  • We have a list of ip addresses which we flag as dangerous.
  • We have incoming log files which sometimes contain these ip addresses.
  • We would like logstash to read this list and compare it with the incoming logs.

I know it is possible to make an if statement with a hardcoded array of the big list of ip addresses which is then compared with grokked log fields "src_ip" or "dst_ip", but this would mean i have to do a daily check whether the big array list has changed or not.

tl;dr: can logstash read an external file list, and let me reach this file list with a variable within the logstash configuration file?

Thanks!

2

Look at the translate plugin.

3

Hi Christian,

Thanks for the mind up :slightly_smiling:

I have read upon this documentation, but am not really understanding how I can implement this for my use case. Could you explain this is in examples in how I can use this?

I did try it, but it doesn't work for me.
in my logstash configuration file I added the following under "filter".

translate {
dictionary_path => "/var/log/otx/list"
refresh_interval => "60"
destination => "detected"
field => "dst_ip"
}

/var/log/otx/list consists of ip addresses like this:
"1.2.3.4": "unsafe"
"2.3.4.5": "unsafe"

I don't really get which index logstash will read the 'field => "dst_ip"' and whether it will write "unsafe" under field 'destination => "detected"' in which index?

After this it should check whether the field "detected" has "unsafe".. or is my whole logic just wrong?

I hope someone can explain the whole process of using translate to do this use case.

Thanks!

4

Just an update for anyone else trying this.

I have it working, but seems like there is something not working for this plugin. (could be my own configuration problem though). But the problem is that it doesn't seem to translate my "destination", but it does add a tag (which is enough for me).

I debugged with "dictionary" (hence the #) and when it kinda worked, i went on to "dictionary_path".
I spoke to someone on irc who said override is needed for translation to overwrite a "destination" but it didn't work for me. I put it on false just in case.

            translate {
                    dictionary_path => "/etc/logstash/otx.list"
                    #dictionary => [ "1.2.3.4", "unsafe" ]
                    add_tag => [ "unsafe" ]
                    field => "dst_ip"
                    override => "false"
            }

the otx file is the same as posted in last post.

Hope this helps for anyone having the same configuration problems.