Hello,
For some time now, we are using ES, Logstash and Kibana with filebeats parsing application logs from different servers. Recently we configured filebeat for the first time using a module.
I followed the instructions regarding "ingest pipelines" (https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html) and everything seems to be configured as expected after using filebeat setup -e -E {Our ES connection}.
But, when filebeat sends some data from the Apache log, we get a WARN in Logstash:
[2020-08-25T14:02:52,918][WARN ][logstash.outputs.elasticsearch][kmt][39c9592b405f52d4dd8d6cdc76fdabe1c39a1f1aaae75361e3838d54c0ad7987] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.9.0-2020.08.25", :routing=>nil, :_type=>"_doc", :pipeline=>"filebeat-7.9.0-apache-access-pipeline"}, #<LogStash::Event:0x6fdbee9f>], :response=>{"index"=>{"_index"=>"filebeat-7.9.0-2020.08.25", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [_doc]: Mapping definition for [source] has unsupported parameters: [properties : {nat={properties={port={type=long}, ip={type=ip}}}, address={ignore_above=1024, type=keyword}, top_level_domain={ignore_above=1024, type=keyword}, ip={type=ip}, mac={ignore_above=1024, type=keyword}, packets={type=long}, geo={properties={continent_name={ignore_above=1024, type=keyword}, region_iso_code={ignore_above=1024, type=keyword}, city_name={ignore_above=1024, type=keyword}, country_iso_code={ignore_above=1024, type=keyword}, country_name={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, region_name={ignore_above=1024, type=keyword}, location={type=geo_point}}}, as={properties={number={type=long}, organization={properties={name={ignore_above=1024, type=keyword, fields={text={norms=false, type=text}}}}}}}, registered_domain={ignore_above=1024, type=keyword}, port={type=long}, service={properties={name={ignore_above=1024, type=keyword}}}, bytes={type=long}, domain={ignore_above=1024, type=keyword}, user={properties={full_name={ignore_above=1024, fields={text={norms=false, type=text}}, type=keyword}, domain={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword, fields={text={norms=false, type=text}}}, id={ignore_above=1024, type=keyword}, email={ignore_above=1024, type=keyword}, hash={ignore_above=1024, type=keyword}, group={properties={domain={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}}}}}}]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"Mapping definition for [source] has unsupported parameters: [properties : {nat={properties={port={type=long}, ip={type=ip}}}, address={ignore_above=1024, type=keyword}, top_level_domain={ignore_above=1024, type=keyword}, ip={type=ip}, mac={ignore_above=1024, type=keyword}, packets={type=long}, geo={properties={continent_name={ignore_above=1024, type=keyword}, region_iso_code={ignore_above=1024, type=keyword}, city_name={ignore_above=1024, type=keyword}, country_iso_code={ignore_above=1024, type=keyword}, country_name={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, region_name={ignore_above=1024, type=keyword}, location={type=geo_point}}}, as={properties={number={type=long}, organization={properties={name={ignore_above=1024, type=keyword, fields={text={norms=false, type=text}}}}}}}, registered_domain={ignore_above=1024, type=keyword}, port={type=long}, service={properties={name={ignore_above=1024, type=keyword}}}, bytes={type=long}, domain={ignore_above=1024, type=keyword}, user={properties={full_name={ignore_above=1024, fields={text={norms=false, type=text}}, type=keyword}, domain={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword, fields={text={norms=false, type=text}}}, id={ignore_above=1024, type=keyword}, email={ignore_above=1024, type=keyword}, hash={ignore_above=1024, type=keyword}, group={properties={domain={ignore_above=1024, type=keyword}, name={ignore_above=1024, type=keyword}, id={ignore_above=1024, type=keyword}}}}}}]"}}}}}
The index has not been created, and I don't understand what parameters are wrong, as the error message is: Failed to parse mapping [_doc]: Mapping definition for [source] has unsupported parameters:
I updated everything (ES, Logstash, Kibana and this Filebeat instance) to version 7.9.0, and every other filebeat input is working fine, even on this filebeat instance there is an input sending the parsed data to Logstash as usual.
Has someone an idea what the problem could be?