ES create index fails when using more than one geoip

Elastic Stack Elasticsearch
1

Starting out with ELK stack, initially to work with Cisco ASA firewall logs. It's been a bit of a learning curve regarding the mapping template config, I finally managed to get the IP addresses mapped to type:ip (after finding that logstash config output section requires "template_overwrite => true" !)

Now I've got geoip (sort of) working, but having problems doing geoip for both the source ip address and destination ip address.
I'm running
elasticsearch 1.7.1 build[b88f43f/2015-07-29T09:54:16Z]
logstash 1.5.4

It all works fine when doing a geoip for either src OR dst ip adddress.
But when I define both the src_geoip AND dst_geoip in the template like this...

<snipped>
       "properties" : {
         "@timestamp": { "type": "date", "doc_values" : true },
         "@version": { "type": "string", "index": "not_analyzed" },
         "src_ip"  : { "type": "ip", "doc_values" : true, "index": "not_analyzed" },
         "dst_ip"  : { "type": "ip", "doc_values" : true, "index": "not_analyzed" },
         "dst_geoip"  : {
           "type" : "object",
           "dynamic": true,
           "properties" : {
             "location" : { "type" : "geo_point", "doc_values" : true },
             "ip": { "type": "ip", "doc_values" : true, "index" : "not_analyzed" },
             "latitude" : { "type" : "float", "doc_values" : true },
             "longitude" : { "type" : "float", "doc_values" : true }
             }
           }
         },
         "src_geoip"  : {
           "type" : "object",
           "dynamic": true,
           "properties" : {
             "location" : { "type" : "geo_point", "doc_values" : true },
             "ip": { "type": "ip", "doc_values" : true, "index" : "not_analyzed" },
             "latitude" : { "type" : "float", "doc_values" : true },
             "longitude" : { "type" : "float", "doc_values" : true }
             }
           }
         }
       }

From logstash filter config...

   geoip {
        database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
        source => "src_ip"
        target => "src_geoip"
   }
   geoip {
        database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
        source => "dst_ip"
        target => "dst_geoip"
   }

ES fails to create the index...

[2015-09-09 10:27:18,068][DEBUG][action.admin.indices.create] [Will o' the Wisp] [logstash-2015.08.15] failed to create
org.elasticsearch.index.mapper.MapperParsingException: mapping [_default_]
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:382)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:196)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:162)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Root type mapping not empty after parsing! Remaining fields:   [src_geoip : {dynamic=true, type=object, properties={ip={index=not_analyzed, type=ip, doc_values=true}, latitude={type=float, doc_values=true}, location={type=geo_point, doc_values=true}, longitude={type=float, doc_values=true}}}]
	at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:278)
	at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:192)
	at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:177)
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:294)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:379)
	... 6 more

I have confirmed that the mapping template is being loaded OK in logstash startup.
I'm wondering if there might be some kind of name clash between both sets of geoip data?

Thanks in advance for any pointers. Rgds, Graeme

2

Generally it'll go that because your mapping is incorrect somewhere.

Can we see the entire thing?

3

Thanks Mark, my mapping template below. Rgds, Graeme

# cat /etc/logstash/templates/elasticsearch-template-custom.json
{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true, "ignore_above" : 256}
               }
           }
         }
       }, {
         "float_fields" : {
           "match" : "*",
           "match_mapping_type" : "float",
           "mapping" : { "type" : "float", "doc_values" : true }
         }
       }, {
         "double_fields" : {
           "match" : "*",
           "match_mapping_type" : "double",
           "mapping" : { "type" : "double", "doc_values" : true }
         }
       }, {
         "byte_fields" : {
           "match" : "*",
           "match_mapping_type" : "byte",
           "mapping" : { "type" : "byte", "doc_values" : true }
         }
       }, {
         "short_fields" : {
           "match" : "*",
           "match_mapping_type" : "short",
           "mapping" : { "type" : "short", "doc_values" : true }
         }
       }, {
         "integer_fields" : {
           "match" : "*",
           "match_mapping_type" : "integer",
           "mapping" : { "type" : "integer", "doc_values" : true }
         }
       }, {
         "long_fields" : {
           "match" : "*",
           "match_mapping_type" : "long",
           "mapping" : { "type" : "long", "doc_values" : true }
         }
       }, {
         "date_fields" : {
           "match" : "*",
           "match_mapping_type" : "date",
           "mapping" : { "type" : "date", "doc_values" : true }
         }
       } ],
       "properties" : {
         "@timestamp": { "type": "date", "doc_values" : true },
         "@version": { "type": "string", "index": "not_analyzed" },
         "src_ip"  : { "type": "ip", "doc_values" : true, "index": "not_analyzed" },
         "dst_ip"  : { "type": "ip", "doc_values" : true, "index": "not_analyzed" },
         "src_geoip"  : {
           "type" : "object",
           "dynamic": true,
           "properties" : {
             "location" : { "type" : "geo_point", "doc_values" : true },
             "ip": { "type": "ip", "doc_values" : true, "index" : "not_analyzed" },
             "latitude" : { "type" : "float", "doc_values" : true },
             "longitude" : { "type" : "float", "doc_values" : true }
             }
           }
         }
       }
    }
  }
}
4

What version are you on?

5

The versions I'm running are:

elasticsearch 1.7.1 build[b88f43f/2015-07-29T09:54:16Z]
logstash 1.5.4

6

Curious if you found a way to get this to work? I'm running into the same problem.

Within the template itself is a wildcard mask legal? So instead of "src_geoip" could we do something along the lines of "geoip*" and then bring the fields in such as "geoip_src" and "geoip_dst"?

TIA.

7

Sorry no unfortunately I never got to the bottom of this.
If you do, could you please let me know :slight_smile: