Cidr Plugin - Warn and skipping data... Missing Data

Elastic Stack Logstash
1

Hello,

I've been playing around with the logstash cidr filter plugin and have been coming into a couple of issues. For the most part, with most of the data sets that are coming in, this filter is working well and is doing the job. BUT, with a small amount of data (seen in the journalctl of logstash) that a warning appears and then a message of "Invaild IP Address, skipping" - example of the log is below:

Log -

[2020-07-28T09:09:03,287][WARN ][logstash.filters.cidr    ][parser1] Invalid IP address, skipping {:address=>"%{[dst_ip]}", :event=>{"src_ip"=>["10.0.0.1", "10.0.0.1"], "dst_interface"=>"inside", "event"=>{"created"=>"Jul 24 05:49:25", "id"=>"166"}, "protocol"=>"TCP", "dst_ip"=>["8.8.8.8", "8.8.8.8"], "tags"=>["taging"], "observer"=>{"type"=>"firewall"}, "dst_port"=>["443", "443"], "openvpn_message"=>"%ASA-6-302013: Built inbound TCP connection 38792873 for outside:10.0.0.1/49543 (10.0.0.1/49543)(LOCAL\\USER) to inside:8.8.8.8/443 (8.8.8.8/443) (USER)", "geo_src"=>{}, "@timestamp"=>2020-07-28T09:07:59.915Z, "direction"=>"inbound", "@version"=>"1", "labels"=>"CONNECTION ", "src_interface"=>"outside", "action"=>"Built", "src_port"=>["49543", "49543"], "src_user"=>"LOCAL\\USER", "connection_id"=>"38792873"}}

The logstash cidr filter in question -

cidr {
      address => [ "%{[dst_ip]}" ]
      network => [ "0.0.0.0/32", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7", "127.0.0.0/8", "::1/128", "169.254.0.0/16", "fe80::/10", "224.0.0.0/4", "ff00::/8", "255.255.255.255/32", "::" ]
      add_field => { "[destination][user][ip]" => "private" }
    }

I have changed the address field in the cidr filter and used the below but still getting the same results.

address => [ "%{[dst_ip]}" ]
address => [ "%{dst_ip}" ]
address => [ "[dst_ip]" ]
address => [ "dst_ip" ]

I have other confs running and have the cidr filter in also and all is running well.
Well with using address => [ "%{[dst_ip]}" ].

When looking around there has been a couple of the same type of issues like :

Saying this issue is closed, but seems to still in a problem?
My Question is that, how come it is working with the majority of the data incoming but only a small bit isn't? Am I missing something?

Am using:
logstash = 7.8.0
cidr filter = 3.1.3v

Thank you!

2

The cidr plugin does not support arrays for the address option. The github issue requesting support for arrays in the address option (#19) was closed when support for arrays in the network option was added. I believe that was an error.

So events that have "dst_ip"=>["8.8.8.8", "8.8.8.8"] will fail. Events that have "dst_ip"=>"8.8.8.8" will work.

3

Hey @Badger,
Thank you very much for your reply!

hmm, ok, right..

So with that then with the reasoning of the filter not accepting arrays, the data will need to be filtered in a different way (different grok pattern) to prevent the array (duplication) of the same IP address. So that the filter in question will be able to perform the needed function for the intended field?

Thank you!

4

Hey,

So on further looking around (and testing the work around mentioned in the previous reference issue). I tried with address => [ "%{[dst_ip][0]}" ] But was still getting the same warning message. That "Invalid IP Address, skipping".

 [2020-07-29T12:04:00,225][WARN ][logstash.filters.cidr    ] Invalid IP address, skipping {:address=>"%{[dst_ip][0]}", :event=>{"observer"=>{"type"=>"firewall"}, "src_ip"=>"8.8.8.8", "dst_port"=>"443", "openvpn_id"=>"%ASA-6-725007", "@version"=>"1", "action"=>"terminated", "REASON"=>"SSL session", "@timestamp"=>2020-07-29T12:02:56.083Z, "tags"=>["tagging"], "event"=>{"created"=>"Jul 25 08:03:05", "id"=>"166"}, "openvpn_message"=>"%ASA-6-725007: SSL session with client outside:8.8.8.8/50520 to 10.0.0.1/443 terminated", "geo_src"=>{}, "labels"=>"5508VPN ", "dst_ip"=>"10.0.0.1", "src_port"=>"50520"}}

As mentioned before that most of the data sets that are in play around being able to be filtered correctly but only a selected view are showing the warning and therefore are unable to be seen when presented in Kibana.

Using this grok pattern to help filter Firewall Grok Patterns.

When it comes to it, I am able to view events such as: %ASA-6-725002, %ASA-6-725001, %ASA-6-7250016 etc... but I an Unable to view events as such: %ASA-6-302013, %ASA-6-302014, %ASA-6-302015, %ASA-6-302016 etc.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.