7.0.1: template with geo_point + geoip processor in pipeline + bulk = crash

Elastic Stack Elasticsearch
1

I'm encountering a similar issue as:

I'm sending logs from kubernetes using fluent-bit to an ES pipeline with a geoip processor.

PUT _ingest/pipeline/geoip
{
  "description": "Ingest nginx-ingress logs",
  "processors": [
    {
      "geoip": {
        "field": "real_ip",
        "target_field": "geoip",
        "ignore_missing": true
      }
    }
  ]
}

It works very well (geoip field is correctly populated)... until I want the coordinates to be a geo_point (sounds familiar, huh?) and add this template:

PUT _template/my_template
{
  "index_patterns": [
    "ingress-*"
  ],
    "mappings" : {
      "properties" : {
        "geo" : {
          "properties" : {
            "location" : {
              "type": "geo_point"
            }
          }
        }
      }
    }
}

Of course I deleted any existing index.

The index is created but no document is added and elasticsearch complains that:

java.lang.IllegalArgumentException: mapper [geo.location] of different type, current_type [geo_point], merged_type [ObjectMapper]

Interestingly it seems to work if create documents from the console.
The issue seems to be related to the fact this is a fresh index by an incoming bulk of documents.

(Almost) full log:
{"type": "server", "timestamp": "2019-05-21T10:42:54,621+0000", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "qCCc98YYReufNd6ZM6wuag", "node.id": "Qn7HpDdJQhKRMnQaEHlfdA", "message": "[ingress-2019.05.21][0] failed to execute bulk item (index) index {[ingress-2019.05.21][flb_type][d155bb33-0bc5-830b-d629-dab72faaae18], source[{"real_ip":"88.181.202.2","referer":"https://kibana.someapp.app/app/kibana\",\"agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36","k8_docker_id":"8697f798cb6a220e2989ef8176ee0d5e1607d65b3e66bfab9076af09d7300b8b","code":"200","upstream_addr":"10.244.3.216:5601","k8_container_name":"nginx-ingress-controller","k8_labels":{"app":"nginx-ingress","component":"controller","release":"nginx-ingress","pod-template-hash":"696d44fbbd"},"k8_host":"general-f6ez","geo":{"continent_name":"Europe","region_iso_code":"FR-77","city_name":"Redacted City","country_iso_code":"FR","region_name":"Seine-et-Marne","location":{"lon":2.5715,"lat":48.3181}},"path":"/elasticsearch/_msearch?rest_total_hits_as_int=trueu0026ignore_throttled=true","upstream_status":"200","k8_pod_id":"9a84914a-77db-11e9-ab45-56d79445c822","request_time":"0.263","stream":"stdout","host":"88.181.202.2","k8_pod_name":"nginx-ingress-controller-696d44fbbd-f6x9p","method":"POST","last":"8f346c238c821ef1348e68ddf1dde2c7","k8_annotations":{"fluentbit_io/parser":"k8s-nginx-ingress"},"upstream_response_length":"210","flb-key":"ingress","message":"88.181.202.2 - [88.181.202.2] - admin [20/May/2019:16:54:29 +0000] \"POST /elasticsearch/_msearch?rest_total_hits_as_int=trueu0026ignore_throttled=true HTTP/2.0\" 200 210 \"https://kibana.someapp.app/app/kibana\\\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36\" 1074 0.263 [monitoring-kibana-kibana-5601] 10.244.3.216:5601 210 0.260 200 8f346c238c821ef1348e68ddf1dde2c7\n","size":"210","k8_namespace_name":"nginx-ingress","request_length":"1074","proxy_upstream_name":"monitoring-kibana-kibana-5601","upstream_response_time":"0.260","time":"2019-05-20T16:54:29.422062755Z","user":"admin","@ts":"2019-05-21T10:42:07.000Z"}]}" ,
"stacktrace": ["java.lang.IllegalArgumentException: mapper [geo.location] of different type, current_type [geo_point], merged_type [ObjectMapper]",
"at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:330) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.GeoPointFieldMapper.doMerge(GeoPointFieldMapper.java:193) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:317) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:52) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:472) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:443) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:47) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:472) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:276) ~[elasticsearch-7.0.1.jar:7.0.1]",

2

Hi @dperetti. There are several discrepancies in the code that you supplied and the logs that you showed. The code seems to work with _doc type the logs indicate flb_type, there is also difference between geoip field in the ingest node configuration and geo type that is used everywhere else. I tried to piece together a reproduction, but something just doesn't add up. This is what I have got so far and it works on 7.0.1:

PUT /_ingest/pipeline/geoip
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "real_ip",
        "target_field": "geoip",
        "ignore_missing": true
      }
    }
  ]
}

PUT _template/my_template
{
  "index_patterns": [
    "test-*"
  ],
  "mappings": {
    "properties": {
      "geoip": {
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
}

POST test-123/_bulk?pipeline=geoip
{"index": {}}
{"real_ip": "192.30.253.113","@timestamp":"2019-05-14T04:33:45.455Z"}
{"index": {}}
{"real_ip": "88.181.202.2","@timestamp":"2019-05-14T04:33:45.455Z"}

Could you let me know what I am missing in this repo?

3

Hello, thanks for the help.
The "geo" / "geoip" is a typo in my report, sorry about this.
The target field is really "geo".
As for the _doc type, that might be it! Can you elaborate ? What do you mean " The code seems to work with _doc type" ?

4

Your template doesn't specify any document type in the mapping, which means _doc is used. The error:

failed to execute bulk item (index) index {[ingress-2019.05.21][flb_type][d155bb33-0bc5-830b-d629-dab72faaae18]

indicates that you specified type flb_type during indexing. So, if there are no other typos in templates and there are no other templates exists, I don't really see how this error can occur since your template shouldn't be applied tothis type.

5

I'm confused, I thought types were deprecated in ES 7.0 ?

Does that mean that fluentbit should not set a type when invoking the bulk operation ?

6

They are indeed deprecated.

It was just deprecated not removed. So it is still possible to setup type, but you need to set templates accordingly. Because at the moment if you don't mention any type the _doc type is used, but by looking at the error message, the bulk request contains flb_type.

What I am still missing though is why you are not getting an error that during indexing about rejected mapping update. When I try this in my example above:

POST test-123/flt_type/_bulk?pipeline=geoip
{"index": {}}
{"real_ip": "192.30.253.113","@timestamp":"2019-05-14T04:33:45.455Z"}
{"index": {}}
{"real_ip": "88.181.202.2","@timestamp":"2019-05-14T04:33:45.455Z"}

I get the error:

ejecting mapping update to [test-123] as the final mapping would have more than 1 type: [_doc, flt_type]

There is still something very important missing in your description of the issue.

7

That was it! I had overlooked the very first error message which complained about the final mapping having more than 1 type.
I'm now forcing fluentbit to use the "_doc" type and it's now working as expected.
Thank you.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.