Error: illegal_argument_exception", "reason"=> FIELD is not an IP string literal.?

Elastic Stack Logstash
1

I have netflow and ipfix data coming in from logstash. When logstash tries to send this to elasticsearch I get these errors.
I tried to tidy up my code using the alter filter instead of all the ifs' and mutate. I am getting the same error no matter what I do when I either enable the alter filter, or if I enable two of the if filters.
Idealy I want to send both ipv6 and ipv4 to the same flow.field.

One flow should only have either one of ipfix.srcipv4, netflow.srcipv4, ipfix.srcipv6, netflow.srcipv6.

This is my config:

#       alter { coalesce => [ "[flow][Source IP]", "[netflow][ipv4_src_addr]", "[netflow][ipv6_src_addr]", "[ipfix][sourceIPv4Address]", "[ipfix][sourceIPv6Address]" ]         }
        if "[netflow][ipv4_src_addr]" { mutate { add_field => {  "[flow][Source IPv4]" => "%{[netflow][ipv4_src_addr]}" } } }
        if "[netflow][ipv6_src_addr]" { mutate { add_field => {  "[flow][Source IPv6]" => "%{[netflow][ipv6_src_addr]}" } } }
#       if "[ipfix][sourceIPv4Address]" { mutate { add_field => {  "[flow][Source IPv4]" => "%{[ipfix][sourceIPv4Address]}" } } }
#       if "[ipfix][sourceIPv6Address]" { mutate { add_field => {  "[flow][Source IPv6]" => "%{[ipfix][sourceIPv6Address]}" } } }
#       alter { coalesce => [ "[flow][Destination IP]", "[netflow][ipv4_dst_addr]", "[netflow][ipv6_dst_addr]", "[ipfix][destinationIPv4Address]", "[ipfix][destinationIPv6Address]" ]         }
#       if "[netflow][ipv4_dst_addr]" { mutate { add_field => {  "[flow][Destination IP]" => "%{[netflow][ipv4_dst_addr]}" } } }
#       if "[netflow][ipv6_dst_addr]" { mutate { add_field => {  "[flow][Destination IP]" => "%{[netflow][ipv6_dst_addr]}" } } }
#       if "[ipfix][destinationIPv4Address]" { mutate { add_field => {  "[flow][Destination IP]" => "%{[ipfix][destinationIPv4Address]}" } } }
#       if "[ipfix][destinationIPv6Address]" { mutate { add_field => {  "[flow][Destination IP]" => "%{[ipfix][destinationIPv6Address]}" } } }

These are the errors I am getting.

[2017-01-05T11:26:36,193][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netflow-2017.01.05", :_type=>"netflow", :_routing=>nil}, 2017-01-05T10:26:36.000Z 172.23.253.1 %{message}], :response=>{"index"=>{"_index"=>"netflow-2017.01.05", "_type"=>"netflow", "_id"=>"AVluKoAe3zwQr38-kLpd", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [flow.Source IPv6]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'%{[netflow][ipv6_src_addr]}' is not an IP string literal."}}}}}


[2017-01-06T12:15:52,755][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netflow-2017.01.06", :_type=>"netflow", :_routing=>nil}, 2017-01-06T11:15:52.000Z 172.23.253.1 %{message}], :response=>{"index"=>{"_index"=>"netflow-2017.01.06", "_type"=>"netflow", "_id"=>"AVlzffky3zwQr38-k784", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [flow.Source IPv4]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'%{[netflow][ipv4_src_addr]}' is not an IP string literal."}}}}}

My mappings in elasticsearch looks like this:

PUT _template/netflow
{
  "template": "netflow-*",
  "settings": { "index.refresh_interval": "5s" },
  "mappings": {
    "_default_": {
      "_all": { "enabled": false},
      "properties": {
        "@timestamp": { "index": "analyzed", "type": "date"},
        "@version":   { "index": "analyzed", "type": "integer"},
        "host":       { "index": "analyzed", "type": "ip"},
        "flow": {
          "dynamic": true,
          "type": "object",
          "properties" : {
            "IP version": { "index": "analyzed", "type": "integer"},
            "Protocol Name": { "index": "analyzed", "type": "string"},
            "Source IP": { "index": "analyzed", "type": "ip"},
            "Source IPv4": { "index": "analyzed", "type": "ip"},
            "Source IPv6": { "index": "analyzed", "type": "ip"},
            "Destination IP": { "index": "analyzed", "type": "ip"},
            "Source Port": { "index": "analyzed", "type": "integer"},
            "Destination Port": { "index": "analyzed", "type": "integer"},
            "Bytes in": { "index": "analyzed", "type": "long" },
            "Packets in": { "index": "analyzed", "type": "long" }
          }
        },
        "ipfix": {
          "dynamic": true,
          "type": "object",
          "properties" : {
            "version": { "index": "analyzed", "type": "integer" },
            "sourceIPv4Address": { "index": "analyzed", "type": "ip" },
            "destinationIPv4Address": { "index": "analyzed", "type": "ip" },
            "postNATSourceIPv4Address": { "index": "analyzed", "type": "ip" },
            "postNATDestinationIPv4Address": { "index": "analyzed", "type": "ip" },
            "ipNextHopIPv4Address": { "index": "analyzed", "type": "ip" },
            "sourceIPv6Address": { "index": "analyzed", "type": "ip" },
            "destinationIPv6Address": { "index": "analyzed", "type": "ip" },
            "ipNextHopIPv6Address": { "index": "analyzed", "type": "ip" },
           .....
2

There's no point in analysing any of those fields FYI.

%{[netflow][ipv6_src_addr]}' is not an IP string literal.

That is why, looks like that is not being translated correctly.

3

Hello @warkholm

I decoded with rubydebug. If I only enable this code: if "[ipfix][sourceIPv4Address]" { mutate { add_field => { "[flow][Source IPv4]" => "%{[ipfix][sourceIPv4Address]}" } } } it seems to fail when I receive a [netflow][field].

It seem to add [flow][Source IPv4] even when line contains only [netflow][field] and no [ipfix] fields.

Why don't I have to analyze those fields? Where can I read more about it?

{
         "ipfix" => {
          "destinationIPv4Address" => "224.0.0.5",
        "destinationTransportPort" => 0,
              "flowStartSysUpTime" => 347803136,
                  "tcpControlBits" => 0,
                "flowEndSysUpTime" => 347803136,
               "sourceIPv4Address" => "172.24.31.242",
                "ingressInterface" => 0,
                "ipClassOfService" => 192,
                         "version" => 10,
                "packetDeltaCount" => 1,
                       "ipVersion" => 4,
                   "flowDirection" => 0,
              "protocolIdentifier" => 89,
                 "egressInterface" => 0,
                 "octetDeltaCount" => 68,
             "sourceTransportPort" => 0
    },
    "@timestamp" => 2017-01-09T14:01:02.000Z,
      "@version" => "1",
          "host" => "172.23.253.1",
          "type" => "netflow",
          "flow" => {
        "Source IPv4" => "172.24.31.242"
    },
          "tags" => [
        [0] "ipfix"
    ]
}
{
       "netflow" => {
                    "dst_as" => 0,
                   "in_pkts" => 4,
            "first_switched" => "2017-01-09T12:18:21.930Z",
             "ipv4_next_hop" => "0.0.0.0",
               "l4_src_port" => 0,
        "sampling_algorithm" => 0,
                  "in_bytes" => 240,
                  "protocol" => 1,
                 "tcp_flags" => 0,
               "l4_dst_port" => 0,
                    "src_as" => 0,
               "output_snmp" => 0,
                  "dst_mask" => 0,
             "ipv4_dst_addr" => "172.23.253.201",
                   "src_tos" => 0,
                  "src_mask" => 0,
                   "version" => 5,
              "flow_seq_num" => 1416,
              "flow_records" => 5,
             "ipv4_src_addr" => "172.18.79.70",
               "engine_type" => 0,
                 "engine_id" => 0,
                "input_snmp" => 2,
             "last_switched" => "2017-01-09T12:18:25.929Z",
         "sampling_interval" => 0
    },
    "@timestamp" => 2017-01-09T14:01:04.930Z,
      "@version" => "1",
          "host" => "172.23.253.201",
          "type" => "netflow",
          "flow" => {
        "Source IPv4" => "%{[ipfix][sourceIPv4Address]}"
    },
          "tags" => [
        [0] "netflow"
    ]
}
4

Im not sure how this works. If I only enable these two rules,

if "[ipfix][sourceIPv4Address]" { mutate { add_field => {  "[flow][Source IPv4]" => "%{[ipfix][sourceIPv4Address]}" } } }
if "[ipfix][sourceIPv6Address]" { mutate { add_field => {  "[flow][Source IPv6]" => "%{[ipfix][sourceIPv6Address]}" } } }

I am still getting error

[2017-01-10T09:13:03,507][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"flow-2017.01.10", :_type=>"flow", :_routing=>nil}, 2017-01-10T08:13:03.000Z 172.23.253.1 %{message}], :response=>{"index"=>{"_index"=>"flow-2017.01.10", "_type"=>"flow", "_id"=>"AVmHcAiR3zwQr38-mDOS", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [flow.Source IPv6]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'%{[ipfix][sourceIPv6Address]}' is not an IP string literal."}}}}}
[2017-01-10T09:13:03,805][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"flow-2017.01.10", :_type=>"flow", :_routing=>nil}, 2017-01-10T08:13:03.000Z 172.23.253.1 %{message}], :response=>{"index"=>{"_index"=>"flow-2017.01.10", "_type"=>"flow", "_id"=>"AVmHcAm73zwQr38-mDOT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [flow.Source IPv6]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'%{[ipfix][sourceIPv6Address]}' is not an IP string literal."}}}}}

It seems like logstash is matching the if "[ipfix][sourceIPv6Address] code even though that field never exists in the event. Then it tries to send the value of [ipfix][sourceIPv6Address] elasticsearch. Elasticsearch rejects it because the value is %{[ipfix][sourceIPv4Address]} ?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.