### Author:  [@JNWL](https://discuss.elastic.co/u/JNWL)
#### Post date: [September 4, 2023, 12:37pm UTC](https://discuss.elastic.co/t/filebeat-setup-error-couldnt-load-template/342268/1 "2023-09-04T12:37:47Z")
Hi All
Years ago I set up a Zeek host with filebeat shipping logs to ELK, this worked fine...
Trying to replicate it on a new host years later, and a lot has changed! I'm getting the below error:
- sudo filebeat setup --index-management -E output..logstash..enabled=false -E 'output..elasticsearch..hosts=["\:443"]'\*
_ILM policy loading not enabled._
_Exiting: error loading template: failed to load template: couldn't load template: 400 Bad Request: {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid"}],"type":"illegal\_argument\_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid","caused\_by":{"type":"illegal\_argument\_exception","reason":"[index\_template] unknown field [data\_stream]"}},"status":400}. Response body: {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid"}],"type":"illegal\_argument\_exception","reason":"composable template [filebeat-8.9.1] template after composition is invalid","caused\_by":{"type":"illegal\_argument\_exception","reason":"[index\_template] unknown field [data\_stream]"}},"status":400_
I'm really stuck here, I can only find templates called filebeat-template.json, filebeat-templatees2x.json and filebeat-templatees6x.json
I'm not really sure what the problem is here, any pointers where to look would be greatly appreciated. For reference I'm using Amazon Linux 2023.
Thanks 🙂
---
### Author:  [@JNWL](https://discuss.elastic.co/u/JNWL)
#### Post date: [September 4, 2023, 2:37pm UTC](https://discuss.elastic.co/t/filebeat-setup-error-couldnt-load-template/342268/2 "2023-09-04T14:37:08Z")
Ok, so I discovered that the version of Elastic was 7.10 so downgraded Filebeat to 7.15,
I'm now stuck with this error:
Exiting: error loading template: failed to load template: couldn't load template: 400 Bad Request: {"error":{"root\_cause":[{"type":"mapper\_parsing\_exception","reason":"No handler for type [flattened] declared on field [metadata]"}],"type":"mapper\_parsing\_exception","reason":"Failed to parse mapping [\_doc]: No handler for type [flattened] declared on field [metadata]","caused\_by":{"type":"mapper\_parsing\_exception","reason":"No handler for type [flattened] declared on field [metadata]"}},"status":400}. Response body: {"error":{"root\_cause":[{"type":"mapper\_parsing\_exception","reason":"No handler for type [flattened] declared on field [metadata]"}],"type":"mapper\_parsing\_exception","reason":"Failed to parse mapping [\_doc]: No handler for type [flattened] declared on field [metadata]","caused\_by":{"type":"mapper\_parsing\_exception","reason":"No handler for type [flattened] declared on field [metadata]"}},"status":400}
I've found other groups that suggested using AWS Elasticsearch causes this and to add these to the filebeat.yml:
setup.ilm.enabled\*\*: false
setup.pack.security.enabled\*\*: false
setup.xpack.graph.enabled\*\*: false
setup.xpack.watcher.enabled\*\*: false
setup.xpack.monitoring.enabled\*\*: false
setup.xpack.reporting.enabled\*\*: false
But I still have this issue.
Details from Elasticsearch host:
"version" : {
"number" : "7.10.2",
"build\_flavor" : "oss",
"build\_type" : "tar",
"build\_hash" : "unknown",
"build\_date" : "2023-01-09T08:02:01.090523Z",
"build\_snapshot" : false,
"lucene\_version" : "8.7.0",
"minimum\_wire\_compatibility\_version" : "6.8.0",
"minimum\_index\_compatibility\_version" : "6.0.0-beta1"
---
### Author:  [@JNWL](https://discuss.elastic.co/u/JNWL)
#### Post date: [September 8, 2023, 10:53am UTC](https://discuss.elastic.co/t/filebeat-setup-error-couldnt-load-template/342268/3 "2023-09-08T10:53:00Z")
Ok, finally getting somewhere, templates loaded after the dev decided to inform me that they send logs only via logstash...
Now I'm getting this:
```auto
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Failed with result 'exit-code'.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: Stopped filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch..
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Start request repeated too quickly.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Failed with result 'exit-code'.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: Failed to start filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch..
[ec2-user@i-03d3c5d8ff13408bf ~]$ sudo filebeat config /etc/filebeat/filebeat.yml
Error: unknown command "config" for "filebeat"
Run 'filebeat --help' for usage.
[ec2-user@i-03d3c5d8ff13408bf ~]$ sudo filebeat test config /etc/filebeat/filebeat.yml
Config OK
[ec2-user@i-03d3c5d8ff13408bf ~]$ journalctl -xe -u filebeat
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: net/http.(*persistConn).writeLoop(0xc0006d4000)
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: /usr/local/go/src/net/http/transport.go:2382 +0xf9
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: created by net/http.(*Transport).dialConn
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: /usr/local/go/src/net/http/transport.go:1744 +0xc9c
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: goroutine 147 [select]:
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: net/http.(*persistConn).writeLoop(0xc0005f6000)
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: /usr/local/go/src/net/http/transport.go:2382 +0xf9
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: created by net/http.(*Transport).dialConn
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: /usr/local/go/src/net/http/transport.go:1744 +0xc9c
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rax 0x0
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rbx 0x7feacffff640
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rcx 0x7feb33ca154c
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rdx 0x6
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rdi 0x2858a6
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rsi 0x2858b5
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rbp 0x2858b5
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rsp 0x7feacfffe790
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r8 0x7feacfffe860
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r9 0x7feb33db14e0
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r10 0x8
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r11 0x246
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
â–‘â–‘ Subject: Unit process exited
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
â–‘â–‘
â–‘â–‘ An ExecStart= process belonging to unit filebeat.service has exited.
â–‘â–‘
â–‘â–‘ The process' exit code is 'exited' and its exit status is 2.
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r12 0x6
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r13 0x0
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r14 0x55addca017f8
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: r15 0x0
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rip 0x7feb33ca154c
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: rflags 0x246
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: cs 0x33
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: fs 0x0
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com filebeat[2644134]: gs 0x0
Sep 08 10:49:43 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Failed with result 'exit-code'.
â–‘â–‘ Subject: Unit failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
â–‘â–‘
â–‘â–‘ The unit filebeat.service has entered the 'failed' state with result 'exit-code'.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
â–‘â–‘ Subject: Automatic restarting of a unit has been scheduled
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
â–‘â–‘
â–‘â–‘ Automatic restarting of the unit filebeat.service has been scheduled, as the result for
â–‘â–‘ the configured Restart= setting for the unit.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: Stopped filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch..
â–‘â–‘ Subject: A stop job for unit filebeat.service has finished
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
â–‘â–‘
â–‘â–‘ A stop job for unit filebeat.service has finished.
â–‘â–‘
â–‘â–‘ The job identifier is 2487595 and the job result is done.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Start request repeated too quickly.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: filebeat.service: Failed with result 'exit-code'.
â–‘â–‘ Subject: Unit failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
â–‘â–‘
â–‘â–‘ The unit filebeat.service has entered the 'failed' state with result 'exit-code'.
Sep 08 10:49:44 i-03d3c5d8ff13408bf.matooma.com systemd[1]: Failed to start filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch..
â–‘â–‘ Subject: A start job for unit filebeat.service has failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
â–‘â–‘
â–‘â–‘ A start job for unit filebeat.service has finished with a failure.
â–‘â–‘
â–‘â–‘ The job identifier is 2487595 and the job result is failed.
```
---
### Author:  [@JNWL](https://discuss.elastic.co/u/JNWL)
#### Post date: [September 8, 2023, 11:02am UTC](https://discuss.elastic.co/t/filebeat-setup-error-couldnt-load-template/342268/4 "2023-09-08T11:02:43Z")
Additionally this is the filebeat.yml:
```auto
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
reload.period: 300s
setup.template.settings:
index.number_of_shards: 1
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: Zeek-IDS
# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["IDS", "Zeek"]
setup.dashboards.enabled: true
output.logstash:
# The Logstash hosts
hosts: ["ourhost:5044"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 6, 2023, 1:03pm UTC](https://discuss.elastic.co/t/filebeat-setup-error-couldnt-load-template/342268/5 "2023-10-06T13:03:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.