Filebeat 7.x compatible with ES 6.x?

Elastic Stack Beats
1

I'm doing a proof of concept with filebeats writing to an existing Elasticsearch 6.4.0 cluster.
I find that I'm unable to write any filebeat data due to the error "Failed to parse mapping [doc]: Mapping definition for [host] has unsupported parameters:

My config is

filebeat.inputs:
- type: stdin
  enabled: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.template.enabled: true

output.console:
  enabled: false
  pretty: true

output.elasticsearch:
  hosts: ["example.com:9200"]
  enabled: true

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

The error message is:

{
    Content: beat.Event {
        Timestamp: time.Time {
            wall: 0xbf54c1b936b923dd,
            ext: 11154020174,
            loc: ( * time.Location)(0x30d3080)
        },
        Meta: common.MapStr(nil),
        Fields: common.MapStr {.....
            "message": "foobar"
        },
        Private: interface {}(nil),
        TimeSeries: false
    },
    Flags: 0x1
}(status = 400): {
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [doc]: Mapping definition for [host] has unsupported parameters:  [properties : {geo={properties={region_iso_code={ignore_above=1024, type=keyword}, continent_name={ignore_above=1024, type=keyword}, city_name={ignore_above=1024, type=keyword}, country_iso_code={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, country_name={ignore_above=1024, type=keyword}, region_name={ignore_above=1024, type=keyword}, location={type=geo_point}}}, hostname={ignore_above=1024, type=keyword}, os={properties={build={ignore_above=1024, type=keyword}, kernel={ignore_above=1024, type=keyword}, codename={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, family={ignore_above=1024, type=keyword}, version={ignore_above=1024, type=keyword}, platform={ignore_above=1024, type=keyword}, full={ignore_above=1024, type=keyword}}}, ip={type=ip}, containerized={type=boolean}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}, type={ignore_above=1024, type=keyword}, user={properties={full_name={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}, email={ignore_above=1024, type=keyword}, hash={ignore_above=1024, type=keyword}, group={properties={name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}}}}}, mac={ignore_above=1024, type=keyword}, architecture={ignore_above=1024, type=keyword}}]",
    "caused_by": {
        "type": "mapper_parsing_exception",
        "reason": "Mapping definition for [host] has unsupported parameters:  [properties : {geo={properties={region_iso_code={ignore_above=1024, type=keyword}, continent_name={ignore_above=1024, type=keyword}, city_name={ignore_above=1024, type=keyword}, country_iso_code={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, country_name={ignore_above=1024, type=keyword}, region_name={ignore_above=1024, type=keyword}, location={type=geo_point}}}, hostname={ignore_above=1024, type=keyword}, os={properties={build={ignore_above=1024, type=keyword}, kernel={ignore_above=1024, type=keyword}, codename={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, family={ignore_above=1024, type=keyword}, version={ignore_above=1024, type=keyword}, platform={ignore_above=1024, type=keyword}, full={ignore_above=1024, type=keyword}}}, ip={type=ip}, containerized={type=boolean}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}, type={ignore_above=1024, type=keyword}, user={properties={full_name={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}, email={ignore_above=1024, type=keyword}, hash={ignore_above=1024, type=keyword}, group={properties={name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}}}}}, mac={ignore_above=1024, type=keyword}, architecture={ignore_above=1024, type=keyword}}]"
    }
}

I'm new to filebeat, but I understand that filebeat will automatically create the index in elasticsearch.

I see the index exists in elasticsearch

curl -s localhost:9200/_cat/indices?v  |grep beat
green  open   filebeat-7.3.1-2019.09.04       xxxxxxxxxx   6   1          0            0        3kb          1.5kb

I downgraded filebeat to 6.8.3, and cleared the template from elasticsearch

curl -XDELETE 'http://localhost:9200/filebeat-*'

I get the same error

Cannot index event publisher.Event {
    ....
 {
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [doc]: Mapping definition for [host] has unsupported parameters:  [properties : {hostname={path=beat.hostname, type=alias}, os={properties={build={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, family={ignore_above=1024, type=keyword}, version={ignore_above=1024, type=keyword}, platform={ignore_above=1024, type=keyword}}}, containerized={type=boolean}, ip={type=ip}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}, mac={ignore_above=1024, type=keyword}, architecture={ignore_above=1024, type=keyword}}]",
    "caused_by": {
        "type": "mapper_parsing_exception",
        "reason": "Mapping definition for [host] has unsupported parameters:  [properties : {hostname={path=beat.hostname, type=alias}, os={properties={build={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, family={ignore_above=1024, type=keyword}, version={ignore_above=1024, type=keyword}, platform={ignore_above=1024, type=keyword}}}, containerized={type=boolean}, ip={type=ip}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}, mac={ignore_above=1024, type=keyword}, architecture={ignore_above=1024, type=keyword}}]"
    }
}

What am I missing? Why can't I write to elasticsearch?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.