I am using filebeat version 7.17.22.
At the start, my filebeat works and sends data to Elasticsearch, which can be seen using Kibana. It sends data to the index.
I have been trying to setup a datastream, so I did so by creating the datastream with the default index pattern filebeat created using the below command (learn from here)
PUT _data_stream/filebeat-7.17.22-
Then, I tried to aim my filebeat output to this datastream by putting the output.elasticsearch.index
as filebeat-7.17.22-
.
However, I got the error:
Exiting: error loading template: failed to load template: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"composable template [filebeat-7.17.22] with index patterns [filebeat-7.17.22-*], priority [150] and no data stream configuration would cause data streams [filebeat-7.17.22-] to no longer match a data stream template"}]
I tried searching it up online but couldn't really find anything that solved the issue.
My filebeat.yml files:
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
paths:
- /path/to/logs/*.txt
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.dashboards.enabled: false
setup.kibana:
host: "http://localhost:5601"
username: "USERNAMEHERE"
password: "PASSWORDHERE"
output.elasticsearch:
hosts: ["https://localhost:9200/"]
preset: balanced
protocol: "https"
username: "USERNAMEHERE"
password: "PASSWORDHERE"
pipeline: "filebeat-pipeline"
index: "filebeat-%{[agent.version]}-"
ssl:
enabled: true
certificate_authorities: /path/to/ca_cert
setup.template.enabled: true
setup.template.name: "filebeat-%{[agent.version]}"
setup.template.pattern: "filebeat-%{[agent.version]}-*"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
logging.level: debug
setup.ilm.enabled: false
setup.ilm.check_exists: true
Some links that I have been looking at to troubleshoot the issue: url1, url2, url3
I also tried to migrate the index to a data stream according to Migrate to data stream API, but just got the error:
"type": "illegal_argument_exception",
"reason": "no matching index template found for data stream [filebeat-7.17.22]"
when running the command:
POST /_data_stream/_migrate/filebeat-7.17.22