Years ago I set up a Zeek host with filebeat shipping logs to ELK, this worked fine...
Trying to replicate it on a new host years later, and a lot has changed! I'm getting the below error:
sudo filebeat setup --index-management -E output..logstash..enabled=false -E 'output..elasticsearch..hosts=["<our.host>:443"]'* ILM policy loading not enabled.
Exiting: error loading template: failed to load template: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid"}],"type":"illegal_argument_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid","caused_by":{"type":"illegal_argument_exception","reason":"[index_template] unknown field [data_stream]"}},"status":400}. Response body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid"}],"type":"illegal_argument_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid","caused_by":{"type":"illegal_argument_exception","reason":"[index_template] unknown field [data_stream]"}},"status":400
I'm really stuck here, I can only find templates called filebeat-template.json, filebeat-templatees2x.json and filebeat-templatees6x.json
I'm not really sure what the problem is here, any pointers where to look would be greatly appreciated. For reference I'm using Amazon Linux 2023.
Ok, so I discovered that the version of Elastic was 7.10 so downgraded Filebeat to 7.15,
I'm now stuck with this error:
Exiting: error loading template: failed to load template: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [flattened] declared on field [metadata]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [metadata]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [flattened] declared on field [metadata]"}},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [flattened] declared on field [metadata]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [metadata]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [flattened] declared on field [metadata]"}},"status":400}
I've found other groups that suggested using AWS Elasticsearch causes this and to add these to the filebeat.yml:
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
reload.period: 300s
setup.template.settings:
index.number_of_shards: 1
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: Zeek-IDS
# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["IDS", "Zeek"]
setup.dashboards.enabled: true
output.logstash:
# The Logstash hosts
hosts: ["ourhost:5044"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
