Gsub can't modify "type" : "ip" to fix string literal error entries for non-ip address info

[danno (Dan Farrell)]

I set field mappings for IP address fields to { "type" : "ip" } and things are great until a logging device (Fortigate) doesn't have an IP address and sends N/A.

"error"=>{"type"=>"mapper_parsing_exception"
"reason"=>"failed to parse [assignip]"
"caused_by"=>{"type"=>"illegal_argument_exception"
"reason"=>"'N/A' is not an IP string literal."}}}}
:level=>:warn}

My desire was to replace N/A with 0.0.0.0 via a mutate/gsub logstash filter, so ES would ingest the data without fail. But mutate/gsub will only work on string fields.

Is there a way to get the same effect via another method?

[magnusbaeck (Magnus Bäck)]

But mutate/gsub will only work on string fields.

(gsub actually also works on fields that are arrays of strings.)

The assignip field is a string field in Logstash so gsub will process it just fine.

[danno (Dan Farrell)]

Thank-you for that amazingly quick reply.

You're correct, by default assignip is a string field. But it and about 7 other fields I've manually templated/set to be { "type" : "ip" } so that ES can parse those fields better/more-efficiently. That was actually a suggestion from a tutorial article from your company - ​Little Logstash Lessons: Using Logstash to help create an Elasticsearch mapping template | Elastic Blog

Even though the IP address is an IP to you and me, Elasticsearch will think of it as a string. For it to be more, you have to tell Elasticsearch how to treat the value, and the way to do that is with a mapping. If an IP is mapped as type ip in Elasticsearch, you can do a query, and filter by an IP range—you can even use CIDR notation—to include only IPs within a given subnet.

But if I map these fields as type ip, then I can't use gsub:

Convert a string field by applying a regular expression and a replacement. If the field is not a string, no action will be taken.

At this point I'm thinking I either need to not use type ip on these different fields or perhaps use gsub -> replace.

Am I understanding (you and/or this situation) correctly?

[paz]

Logstash type and ElasticSearch type are 2 different things.

Logstash will send it as string, and then ElasticSearch will analyze that string and break it down as an IP (which is actually an integer internally). But it' still a string until it reaches ES.

Bottom line, ElasticSearch mappings won't block you from gsub-ing a string.

[danno (Dan Farrell)]

That makes so much sense, and thank-you!

Both of you guys are AWESOME!

[magnusbaeck (Magnus Bäck)]

That was actually a suggestion from a tutorial article from your company

To be clear, Elastic isn't my company.

[danno (Dan Farrell)]

I apologize, and still think you're awesome.

[danno (Dan Farrell)]

So my 'fix' isn't parsing in LS, but isn't crashing LS, either.

I've tried gsub and replace :

     if "N\/A" in ["assignip"] {
        mutate { 
           gsub => ["assignip","N\/A","0\.0\.0\.0"]
          #replace => {"assignip" => "0\.0\.0\.0"}
        }
    }

(I've alternately commented out the gsub statement and commented-in the replace statement).

I'm still getting this specific error-

(only including the interesting information snippet from the full error log entry):

"assignip"=>"N/A", ... "eventstatement"=>"progress IPsec phase 2"}, "type"]}>>], :response=>{"index"=>{"_index"=>"logstash-2017.06.29", "_type"=>"fortigate", "_id"=>"AVz0P_x-xWFZrN1eX51d", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [assignip]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'N/A' is not an IP string literal."}}}}, :level=>:warn}

Any help on this is very appreciated.

[magnusbaeck (Magnus Bäck)]

Have you tried not escaping the forward slashes or the periods?

[danno (Dan Farrell)]

Just tried that on both, including not escaping N/A but escaping the ip address, not escaping anything, and of course, escaping everything.

I understand the explanation that LS will not worry about the type that ES will use for the field but it really appears to be doing so with the error ... that's an LS error entry, not an ES error entry.

I've even tried if "N/A" in ["assignip"] {mutate {remove_field => ["assignip"]}} and the same error occurs.

[danno (Dan Farrell)]

I've solved my issue, it was my fault (of course!). I was already using a mutate above this and I guess it only runs once per nested set of conditionals? I say this because I plopped the gsub statement into the mutate above and it worked!

Basically-

if [message] =~ ".*<d\+>.*"{ kv { source => "message" exclude_keys => [ "type", "subtype" ] add_tag => [ "fortigate-kv" ] } mutate { rename => [ "dst", "dst_ip" ] rename => [ "dstip", "dst_ip" ] rename => [ "dstport", "dst_port" ] rename => [ "devname", "hostsourcename" ] rename => [ "status", "action" ] rename => [ "src", "src_ip" ] rename => [ "srcip", "src_ip" ] rename => [ "zone", "src_intf" ] rename => [ "srcintf", "src_intf" ] rename => [ "srcport", "src_port" ] rename => [ "rcvd", "byte_recieved" ] rename => [ "rcvdbyte", "bytes_recieved" ] rename => [ "sentbyte", "bytes_sent" ] rename => [ "sent", "bytes_sent" ] convert => ["bytes_recieved", "integer"] convert => ["bytes_sent", "integer"] rename => [ "msg", "eventstatement" ] gsub => ["assignip","N/A","0.0.0.0"] remove_field => [ "message" ] } } if "N/A" in ["dst_ip"] { mutate { gsub => [ "dst_ip","N/A","0.0.0.0" ] } }

In this example, the dst_ip mutate filter is effectively ignored (that's where the assignip gsub was) . kv hands off the newly-minted fields to mutate, which does it's work. But another nested conditional that calls upon mutate again will be ignored (apparently).

Thank-you very much for your help, I could not have gotten to this point without it.

[magnusbaeck (Magnus Bäck)]

I understand the explanation that LS will not worry about the type that ES will use for the field but it really appears to be doing so with the error ... that's an LS error entry, not an ES error entry.

Yes, but it's only relaying an ES error message.

In this example, the dst_ip mutate filter is effectively ignored

Probably because you should've used

if "N/A" in [dst_ip] {

instead of

if "N/A" in ["dst_ip"] {

Watch out when using multiple options in the same mutate filter. They're not executed in the order listed (necessarily anyway). If you're doing stuff with different fields it doesn't matter but if you modify the same field in a sequence of operations that will likely fail.

[danno (Dan Farrell)]

Thank-you for this, I will definitely keep it in mind.
And you're right. This works:

if "N/A" in [assignip]{ mutate {gsub => ["assignip","N/A","0.0.0.0"]}}