Mixing up field types error: IP field is not getting added

Elastic Stack Elasticsearch
1

i did a manual mapping in Elasticsearch. but i am getting the below error. Can you please help.

Failed action. {:status=>400, :action=>["index", {:_id=>nil, :index=>"netflow", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x481034c7 @metadata_accessors=#<LogStash::Util::Accessors:0x7b26b09 @store={}, @lut={}>, @cancelled=false, @data={"tag"=>0, "as_src"=>0, "label"=>"DFW1", "ip_src"=>"10.160.10.31", "packets"=>1, "country_ip_dst"=>"", "as_dst"=>0, "peer_ip_src"=>"10.0.11.252", "bytes"=>55, "iface_out"=>30, "iface_in"=>18, "ip_dst"=>"10.160.22.31", "mask_src"=>0, "mask_dst"=>0, "port_src"=>45005, "port_dst"=>15353, "stamp_inserted"=>"2016-07-12 15:48:00", "country_ip_src"=>"", "ip_proto"=>"udp", "stamp_updated"=>"2016-07-12 15:51:10", "@version"=>"1", "@timestamp"=>"2016-07-12T15:54:37.015Z"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6e830327 @store={"tag"=>0, "as_src"=>0, "label"=>"DFW1", "ip_src"=>"10.160.10.31", "packets"=>1, "country_ip_dst"=>"", "as_dst"=>0, "peer_ip_src"=>"10.0.11.252", "bytes"=>55, "iface_out"=>30, "iface_in"=>18, "ip_dst"=>"10.160.22.31", "mask_src"=>0, "mask_dst"=>0, "port_src"=>45005, "port_dst"=>15353, "stamp_inserted"=>"2016-07-12 15:48:00", "country_ip_src"=>"", "ip_proto"=>"udp", "stamp_updated"=>"2016-07-12 15:51:10", "@version"=>"1", "@timestamp"=>"2016-07-12T15:54:37.015Z"}, @lut={"type"=>[{"tag"=>0, "as_src"=>0, "label"=>"DFW1", "ip_src"=>"10.160.10.31", "packets"=>1, "country_ip_dst"=>"", "as_dst"=>0, "peer_ip_src"=>"10.0.11.252", "bytes"=>55, "iface_out"=>30, "iface_in"=>18, "ip_dst"=>"10.160.22.31", "mask_src"=>0, "mask_dst"=>0, "port_src"=>45005, "port_dst"=>15353, "stamp_inserted"=>"2016-07-12 15:48:00", "country_ip_src"=>"", "ip_proto"=>"udp", "stamp_updated"=>"2016-07-12 15:51:10", "@version"=>"1", "@timestamp"=>"2016-07-12T15:54:37.015Z"}, "type"]}>>], :response=>{"create"=>{"index"=>"netflow", "_type"=>"logs", "_id"=>"AVXf0TPx-AQHNLHZG1yY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Mixing up field types: class org.elasticsearch.index.mapper.core.LongFieldMapper$LongFieldType != class org.elasticsearch.index.mapper.ip.IpFieldMapper$IpFieldType on field ip_src"}}}}, :level=>:warn}

Below is my template config:

curl -XGET localhost:9200/template/*?pretty
{
"my_logs" : {
"order" : 1,
"template" : "netflow*",
"settings" : {
"index" : {
"number_of_shards" : "1",
"number_of_replicas" : "0"
}
},
"mappings" : {
"logs" : {
"properties" : {
"ip_proto" : {
"type" : "string"
},
"iface_in" : {
"type" : "long"
},
"iface_out" : {
"type" : "long"
},
"src_host_country" : {
"type" : "geo_point"
},
"peer_ip_src" : {
"type" : "ip"
},
"ip_src" : {
"type" : "ip"
},
"port_dst" : {
"type" : "long"
},
"port_src" : {
"type" : "long"
},
"ip_dst" : {
"type" : "ip"
},
"as_src" : {
"type" : "long"
},
"@timestamp" : {
"type" : "date"
},
"mask_dst" : {
"type" : "ip"
},
"mask_src" : {
"type" : "ip"
},
"dst_host_country" : {
"type" : "geo_point"
},
"as_dst" : {
"type" : "long"
}
}
}
},
"aliases" : { }
}
}

2

Can anyone help to fix this.

3

Is the ip_src field mapped as a long on one of your types?

4

No in template i mentioned it as type IP.. so the mapping should happen based on template right?
sorry i am very new to elastic search

5

Right, so I was wondering that maybe you are putting explicit mappings as well in the same index?

If not, can you provide us with a contained recreation of the issue that we can look into?

6

No.. I am not putting any mapping in the index.
The error is gone now.. but i am not able to change the type field of ip_src and ip_dst to ip. it takes the default value when the index get created.

Here is my logstash.conf

input {
kafka {
zk_connect => "localhost:3181"
topic_id => "pmacct.acct"
}
}
output {
elasticsearch {
hosts => "127.0.0.1"
index => "logs_%{+YYYY_MM_dd}"
manage_template => false
}
}

My template file:

curl -XGET localhost:9200/template/*?pretty
{
"logs" : {
"order" : 0,
"template" : "logs*",
"settings" : {
"index" : {
"number_of_shards" : "1",
"number_of_replicas" : "0",
"mappings" : {
"logs" : {
"properties" : {
"@version" : {
"type" : "integer",
"index" : "analyzed"
},
"netflow" : {
"type" : "object",
"dynamic" : "strict",
"properties" : {
"iface_in" : {
"type" : "long"
},
"ip_proto" : {
"type" : "string"
},
"iface_out" : {
"type" : "long"
},
"ip_src" : {
"type" : "ip"
},
"peer_ip_src" : {
"type" : "ip"
},
"port_dst" : {
"type" : "long"
},
"port_src" : {
"type" : "long"
},
"label" : {
"type" : "string"
},
"ip_dst" : {
"type" : "ip"
},
"packets" : {
"type" : "long"
},
"as_src" : {
"type" : "long"
},
"stamp_updated" : {
"type" : "date"
},
"stamp_inserted" : {
"type" : "date"
},
"bytes" : {
"bytes" : "long"
},
"country_ip_src" : {
"type" : "geo_point"
},
"mask_dst" : {
"type" : "long"
},
"country_ip_dst" : {
"type" : "geo_point"
},
"mask_src" : {
"type" : "long"
},
"tag" : {
"type" : "number"
},
"as_dst" : {
"type" : "long"
}
}
},
"@timestamp" : {
"type" : "date",
"index" : "analyzed"
}
}
}
}
}
},
"mappings" : { },
"aliases" : { }
}
}

Elasticsearch log:

[2016-07-13 17:57:02,249][INFO ][cluster.metadata ] [Barnacle] [logs_2016_07_13] creating index, cause [auto(bulk api)], templates [logs], shards [1]/[0], mappings [logs]
[2016-07-13 17:57:02,263][INFO ][cluster.routing.allocation] [Barnacle] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logs_2016_07_13][0]] ...]).
[2016-07-13 17:57:02,269][INFO ][cluster.metadata ] [Barnacle] [logs_2016_07_13] update_mapping [logs]

7

This is the mapping file which got created automatically.

curl -XGET localhost:9200/logs_2016_07_13/_mappings/?pretty=true
{
"logs_2016_07_13" : {
"mappings" : {
"logs" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "strict_date_optional_time||epoch_millis"
},
"@version" : {
"type" : "string"
},
"as_dst" : {
"type" : "long"
},
"as_src" : {
"type" : "long"
},
"bytes" : {
"type" : "long"
},
"country_ip_dst" : {
"type" : "string"
},
"country_ip_src" : {
"type" : "string"
},
"iface_in" : {
"type" : "long"
},
"iface_out" : {
"type" : "long"
},
"ip_dst" : {
"type" : "string"
},
"ip_proto" : {
"type" : "string"
},
"ip_src" : {
"type" : "string"
},
"label" : {
"type" : "string"
},
"mask_dst" : {
"type" : "long"
},
"mask_src" : {
"type" : "long"
},
"packets" : {
"type" : "long"
},
"peer_ip_src" : {
"type" : "string"
},
"port_dst" : {
"type" : "long"
},
"port_src" : {
"type" : "long"
},
"stamp_inserted" : {
"type" : "string"
},
"stamp_updated" : {
"type" : "string"
},
"tag" : {
"type" : "long"
}
}
}
}
}
}

8

The template is not applied because it looks for index names that start with netflow_ ("template" : "netflow_*") while this index name is logs_2016_07_13.