Filebeat does not insert data into elasticsearch using ILM (node default index)

[Alon_Eldi]

Hi , We have filebeat agents which write data directly into elasticsearch cloud service.

For some reason the filebeat is not writing the data into the index

The filbeat use ILM policy :

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "data-prod"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "data-prod-%{[agent.version]}"
setup.ilm.check_exists: false
setup.ilm.overwrite: true
setup.ilm.policy_file: /etc/filebeat/index.json

/etc/filebeat/index.json

{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "24h",
"max_size": "50gb",
"max_docs": 100000000
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "61d",
"actions": {
"delete": {}
}
}
}
}
}

the data-prod alias have 1 writing index .

"data-prod-2020.04.16-000600" : {
"aliases" : {
"data-prod" : {
"is_write_index" : true
}
}
}

I wrote to the alias from the filebeat machine successfully :

curl -k -u elastic -X PUT "https:///data-prod/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
"name": "John Doe"
}'

So the filebeat agent can write to elasticsearch and to the correct index .

I run filebeat in debug mode :
filebeat -e -d "*" -c /etc/filebeat/filebeat.yml

dont get any errors and output show correct values :

2020-04-22T13:10:56.509Z INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'data-prod' as ILM is enabled.
2020-04-22T13:10:56.509Z INFO elasticsearch/client.go:170 Elasticsearch url: https://:443

Do I miss anything

Thanks

Alon

[shaunak (Shaunak Kashyap)]

Hi @Alon_Eldi, welcome to the Elastic community forums!

Just want to clarify something: is Filebeat not writing data into any Elasticsearch index? Or is it writing data but just not into the data-prod-2020.04.16-000600 index (via the data-prod write alias)?

Shaunak

[Alon_Eldi]

It’s not writing to any index .
And does not show any error in filebeat debug output .

[shaunak (Shaunak Kashyap)]

Okay, thanks. In that case I would try to first narrow down where the problem is happening — Filebeat or Elasticsearch. To do this, could you temporarily disable the Elasticsearch output in Filebeat by setting output.elasticsearch.enabled: false and enable the Console output instead by setting output.console.enabled: true? Then restart Filebeat and let us know if you are seeing any output from Filebeat to your console.

Thanks,

Shaunak