Object mapping issue

Hi,

I am getting following issue in elasticsearch. Object mapping for [ClientIP] tried to parse field [ClientIP] as object. How do I fixed configuration to adjust the mapping ?

[2019-08-19T16:03:30,213][DEBUG][o.e.a.b.TransportShardBulkAction] [9pMb_Kb] [cloudflare-2019-08-19][0] failed to execute bulk item (index) index {[<cloudflare-{2019-08-19||/w{yyyy-MM-dd|UTC}}>][doc][ezrCqWwBC8sWyWQexwgZ], source[n/a, actual length: [2kb], max length: 2kb]}
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [ClientIP] tried to parse field [ClientIP] as object, but found a concrete value
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:360) ~[elasticsearch-6.7.2.jar:6.7.2]

Regards,
Osama

can you share the mapping for that index, you can use filter_path=**.ClientId to filter only for that mapping in order to reduce noise.

Hi,

I have attached the screenshot of below command:

GET _mapping/field/filter_path=**.ClientIP

What version of Elasticsearch are you running?

Did you follow the instructions for Cloudflare integration? You can find the documentation here: https://developers.cloudflare.com/logs/analytics-integrations/elastic/.

It looks like you did not run the install_artifacts.sh script which creates the index template.

I am using ELK 6.7.2 version. I have completely followed creating index template as mentioned in docs

Please paste the output of GET _template/cloudflare. Thanks!

Hi
Below is the output, It was a long output but I have cut it to short:

#! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
{
  "cloudflare" : {
    "order" : 0,
    "index_patterns" : [
      "cloudflare-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "doc" : {
        "properties" : {
          "WAFRuleID" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            }
          },
          "ZoneID" : {
            "type" : "long"
          },
          "OriginResponseTime" : {
            "type" : "long"
          },
          "OriginResponseHTTPExpires" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "type" : "text"
          },
          "RayID" : {
            "type" : "keyword"
          },
          "SecurityLevel" : {
            "type" : "keyword"
          },
          "EdgeResponseCompressionRatio" : {
            "type" : "float"
          },
          "EdgeColoID" : {
            "type" : "long"
          },
          "ClientRequestProtocol" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            }
          },
          "ClientSrcPort" : {
            "type" : "long"
          },
          "EdgeResponseStatus" : {
            "type" : "long"
          },
          "OriginResponseBytes" : {
            "type" : "long"
          },
          "OriginResponseHTTPLastModified" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "type" : "text"
          },
          "ClientSSLProtocol" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "type" : "text"
          },
          "EdgePathingOp" : {
            "type" : "keyword"
          },
          "ClientIP" : {
            "properties" : {
              "timezone" : {
                "type" : "keyword"
              },
              "ip" : {
                "type" : "keyword"
              },
              "latitude" : {
                "type" : "float"
              },
              "continent_code" : {
                "type" : "keyword"
              },
              "city_name" : {
                "type" : "keyword"
              },
              "dma_code" : {
                "type" : "long"
              },
              "country_code2" : {
                "type" : "keyword"
              },
              "country_name" : {
                "type" : "keyword"
              },
              "country_code3" : {
                "type" : "keyword"
              },
              "location" : {
                "type" : "geo_point"
              },
              "region_name" : {
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                },
                "type" : "text"
              },
              "postal_code" : {
                "type" : "keyword"
              },
              "longitude" : {
                "type" : "float"
              },
              "region_code" : {
                "type" : "keyword"
              }
            }
          },
          "CacheTieredFill" : {
            "type" : "boolean"
          },
          "ClientSSLCipher" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "type" : "text"
          },
          "ParentRayID" : {
            "type" : "keyword"
          },
          "EdgeRequestHost" : {
            "type" : "keyword"
          },

        }
      }
    },
    "aliases" : { }
  }
}

The mapping looks correct. Can you paste the output of GET _ingest/pipeline/cloudflare-pipeline-daily?

Also, are you referencing the pipeline when you ingest documents?

Hi, Please find below output for GET _ingest/pipeline/cloudflare-pipeline-daily

{
  "cloudflare-pipeline-daily" : {
    "description" : "Cloudflare Log Pipeline (Daily Indices)",
    "processors" : [
      {
        "date" : {
          "field" : "EdgeStartTimestamp",
          "formats" : [
            "yyyy-MM-dd'T'HH:mm:ssZ",
            "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
          ],
          "timezone" : "UTC",
          "target_field" : "@timestamp"
        }
      },
      {
        "date" : {
          "field" : "EdgeStartTimestamp",
          "formats" : [
            "yyyy-MM-dd'T'HH:mm:ssZ",
            "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
          ],
          "timezone" : "UTC",
          "target_field" : "EdgeStartTimestamp"
        }
      },
      {
        "date_index_name" : {
          "field" : "EdgeStartTimestamp",
          "index_name_prefix" : "cloudflare-",
          "date_rounding" : "d",
          "timezone" : "UTC",
          "date_formats" : [
            "yyyy-MM-dd'T'HH:mm:ssZ",
            "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
          ]
        }
      },
      {
        "geoip" : {
          "field" : "ClientIP",
          "target_field" : "ClientIP",
          "properties" : [
            "ip",
            "country_iso_code",
            "country_name",
            "continent_name",
            "region_iso_code",
            "region_name",
            "city_name",
            "timezone",
            "location"
          ]
        }
      },
      {
        "user_agent" : {
          "field" : "ClientRequestUserAgent",
          "target_field" : "UserAgent"
        }
      }
    ]
  }
}

That looks correct. My guess is that you are not referencing the pipeline when you ingest documents. What are you using to send documents to ES?

I am using AWS lamda function in order to send logs in elasticsearch by follow this link

Here is the mapping details for ClientIP:

 "ClientIP": {
          "properties": {
            "city_name": {
              "type": "keyword"
            },
            "continent_code": {
              "type": "keyword"
            },
            "continent_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code2": {
              "type": "keyword"
            },
            "country_code3": {
              "type": "keyword"
            },
            "country_iso_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_name": {
              "type": "keyword"
            },
            "dma_code": {
              "type": "long"
            },
            "ip": {
              "type": "keyword"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            },
            "postal_code": {
              "type": "keyword"
            },
            "region_code": {
              "type": "keyword"
            },
            "region_iso_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },

Below are the settings:

{
  "settings": {
    "index": {
      "creation_date": "1566778069365",
      "number_of_shards": "1",
      "number_of_replicas": "1",
      "uuid": "TX2kHDbHQGSqB9T7lL5IuA",
      "version": {
        "created": "6070299"
      },
      "provided_name": "<cloudflare-{2019-08-26||/w{yyyy-MM-dd|UTC}}>"
    }
  },
  "defaults": {
    "index": {
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "lifecycle": {
        "name": "",
        "rollover_alias": "",
        "indexing_complete": "false"
      },
      "routing_partition_size": "1",
      "max_docvalue_fields_search": "100",
      "merge": {
        "scheduler": {
          "max_thread_count": "4",
          "auto_throttle": "true",
          "max_merge_count": "9"
        },
        "policy": {
          "reclaim_deletes_weight": "2.0",
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "5gb",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "deletes_pct_allowed": "33.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "5",
      "write": {
        "wait_for_active_shards": "1"
      },
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "depth": {
          "limit": "20"
        },
        "ignore_malformed": "false",
        "total_fields": {
          "limit": "1000"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "false",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "default_field": [
          "*"
        ],
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "priority": "1",
      "codec": "default",
      "max_rescore_window": "10000",
      "max_adjacency_matrix_filters": "100",
      "gc_deletes": "60s",
      "optimize_auto_generated_id": "true",
      "max_ngram_diff": "1",
      "translog": {
        "generation_threshold_size": "64mb",
        "flush_threshold_size": "512mb",
        "sync_interval": "5s",
        "retention": {
          "size": "512mb",
          "age": "12h"
        },
        "durability": "REQUEST"
      },
      "auto_expand_replicas": "false",
      "mapper": {
        "dynamic": "true"
      },
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "data_path": "",
      "highlight": {
        "max_analyzed_offset": "-1"
      },
      "routing": {
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
      },
      "search": {
        "slowlog": {
          "level": "TRACE",
          "threshold": {
            "fetch": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            },
            "query": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          }
        },
        "throttled": "false"
      },
      "fielddata": {
        "cache": "node"
      },
      "default_pipeline": "_none",
      "max_slices_per_scroll": "1024",
      "shard": {
        "check_on_startup": "false"
      },
      "xpack": {
        "watcher": {
          "template": {
            "version": ""
          }
        },
        "version": "",
        "ccr": {
          "following_index": "false"
        }
      },
      "percolator": {
        "map_unmapped_fields_as_text": "false",
        "map_unmapped_fields_as_string": "false"
      },
      "allocation": {
        "max_retries": "5"
      },
      "refresh_interval": "1s",
      "indexing": {
        "slowlog": {
          "reformat": "true",
          "threshold": {
            "index": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          },
          "source": "1000",
          "level": "TRACE"
        }
      },
      "compound_format": "0.1",
      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },
      "max_result_window": "10000",
      "store": {
        "stats_refresh_interval": "10s",
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": []
      },
      "queries": {
        "cache": {
          "enabled": "true"
        }
      },
      "ttl": {
        "disable_purge": "false"
      },
      "warmer": {
        "enabled": "true"
      },
      "max_shingle_diff": "3",
      "query_string": {
        "lenient": "false"
      }
    }
  }
}

Here's the log shown in elasticsearch.log:

[2019-08-30T10:58:45,420][DEBUG][o.e.a.b.TransportShardBulkAction] [9pMb_Kb] [cloudflare-2019-08-26][0] failed to execute bulk item (index) index {[<cloudflare-{2019-08-30||/w{yyyy-MM-dd|UTC}}>][doc][ZHxR4WwBY9LaTT4fuhcl], source[{"WAFRuleID":"","ZoneID":80420011,"OriginResponseTime":233000000,"OriginResponseHTTPExpires":"","RayID":"00116288bdd8074","SecurityLevel":"med","FirewallMatchesSources":[],"EdgeResponseCompressionRatio":4.99,"ClientRequestProtocol":"HTTP/1.1","EdgeColoID":113,"ClientSrcPort":51598,"EdgeResponseStatus":200,"OriginResponseBytes":0,"OriginResponseHTTPLastModified":"","FirewallMatchesActions":[],"ClientSSLProtocol":"TLSv1.2","EdgePathingOp":"wl","CacheTieredFill":false,"ClientIP":"120.64.144.102","ClientSSLCipher":"ECDHE-RSA-AES128-GCM-SHA256","ParentRayID":"00","EdgeRequestHost":"test.abc.com","ClientRequestUserAgent":"Th%20abc/1719 CFNetwork/978.0.7 Darwin/18.6.0","EdgePathingSrc":"macro","EdgeRateLimitID":0,"ClientDeviceType":"desktop","ClientIPClass":"noRecord","WorkerSubrequestCount":0,"OriginSSLProtocol":"unknown","WAFMatchedVar":"","EdgeRateLimitAction":"","EdgeResponseBytes":1815,"ClientRequestURI":"/test/web/v23/get_action","WorkerSubrequest":false,"EdgeStartTimestamp":"2019-08-30T06:51:30.000Z","WAFFlags":"0","ClientRequestHost":"test.abc.com","ClientRequestPath":"/test/web/v23/get_action","WorkerStatus":"unknown","OriginResponseStatus":200,"UserAgent":{"patch":"7","major":"978","minor":"0","os":"Other","name":"CFNetwork","os_name":"Other","device":"iOS-Device"},"CacheCacheStatus":"unknown","OriginIP":"10.10.10.10","ClientASN":328453,"WAFProfile":"unknown","WAFAction":"unknown","FirewallMatchesRuleIDs":[],"ClientCountry":"za","ClientRequestReferer":"","ClientRequestBytes":2178,"WAFRuleMessage":"","EdgeResponseContentType":"application/json","WorkerCPUTime":0,"@timestamp":"2019-08-30T06:51:30.000Z","EdgeColoCode":"CPT","EdgeServerIP":"","EdgePathingStatus":"nr","CacheResponseBytes":5481,"ClientRequestMethod":"POST","CacheResponseStatus":200,"EdgeEndTimestamp":"2019-08-30T06:51:30Z"}]} 
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [ClientIP] tried to parse field [ClientIP] as object, but found a concrete value
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:360) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:616) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:410) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:799) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:775) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370) ~[elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) [elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) [elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) [elasticsearch-6.7.2.jar:6.7.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987) [elasticsearch-6.7.2.jar:6.7.2]
	at 

I have added some dummy values to hide actual information like IP and domains: test.abc.com

@andrew Please look into it, if you need further information please let me know.

Here's the screenshot of index pattern in Kibana:

Hi Again,

Warning "#! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."

I am using elasticsearch 6.7.2 and in v6.8.2 fixed the deprecation warning. LINK

According to elasticsearch. "Not setting include_type_name will result in a deprecation warning. Indices which don’t have an explicit type will use the dummy type name _doc."

Might be it causes the object mapping issue. ? Any recommendation ?

As we have updated the ELK version from 6.7.2 to 6.8.3. But, We still getting same issue ? Please update.

Anyone please update this, How can I rid off this issue ?