Configuration using filebeat with static index name

Hi,

I would like to know how I need to configure filebeat to use an existing index(it's empty). What's the configuration to do so? So far I tried in a helm chart:

 output.elasticsearch:
      hosts: 'https://XXXXXX.com:8443'
      index: 'my-index-filebeat'
      username: "XXXXX"
      password: "YYYYY"
    setup.template:
      name: 'my-index-filebeat'
      overwrite: true
      enabled: true
      pattern: 'my-index-filebeat'

I cannot create index in this elasticsearch server, only use this one.

Thanks in advance

It sounds like you are running into this issue: https://github.com/elastic/beats/issues/11866.

The workaround is to set setup.ilm.enabled: false in your Filebeat configuration. Could you try that and see if the my-index-filebeat template and my-index-filebeat index get created?

Thanks,

Shaunak

Thanks Shanuak,

I tried with that option but unfortunately it doesn't work.

My case it's a little bit uncommon. I need to use the my-index-filebeat index that was created by the elasticsearch admins in my company. I cannot create any index in the elasticsearch server.
What should I do to make it work?

Logs:

2019-10-17T13:56:00.894Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://XXXXXXXXXX.com:8443)) with 4 reconnect attempt(s)
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:191 done
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:168 done
2019-10-17T13:56:00.903Z INFO elasticsearch/client.go:743 Attempting to connect to Elasticsearch version 6.8.2
2019-10-17T13:56:00.914Z INFO template/load.go:169 Existing template will be overwritten, as overwrite is enabled.
2019-10-17T13:56:00.984Z INFO template/load.go:108 Try loading template my-index-filebeat to Elasticsearch

Indexing into an existing index should be fine; I just tested it to make sure.

As a test, could you try to send a Bulk API request, using the same username+password as in your Filebeat configuration, to my-index-filebeat from the command line (using curl or equivalent) on the host where Filebeat is running?

That'll tell us if a) a connection can be made from the Filebeat host to the Elasticsearch cluster and b) if the configured user has sufficient privileges to bulk index documents into my-index-filebeat.

Also, are those all the logs you are seeing for Filebeat? Specifically, are there any log messages showing errors or warnings?

Finally, which version of Filebeat are you using?

Shaunak

As requested:

$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }

curl -s -H "Content-Type: application/x-ndjson" -XPOST https://XXXXXX.com:8443/_bulk --data-binary "@requests" -u XXXXXX; echo
Enter host password for user 'XXXXXX':
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: type is missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: type is missing;"},"status":400}

  • Example Connection:

$ curl -XGET "https://XXXXXXX.com:8443/my-index-filebeat/_search?pretty=true&q=:" -u XXXXXX
Enter host password for user 'XXXXXX':
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" :
}
}

  • I will double check the permissions with our team.

  • Only I have these INFO messages, get stuck in Try loading template ....

  • Filebeat version: 7.3.2

Many Thanks.

Can you try changing this line in requests to:

{ "index" : { "_index" : "test", "_type": "_doc", "_id" : "1" } }

and make the curl request again please?

{"took":9,"errors":true,"items":[{"index":{"_index":"test","_type":"_doc","_id":"1","status":403,"error":{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [XXXXX]"}}}]}

Just a reminder, I cannot create index only using the existing one.

Thanks.

So is test not an existing index? Can you try the bulk request from curl using the same existing index that you would use in your output.elasticsearch.index setting?

Sorry, I didn't changed the name of the index, this is the result with my index:

{"took":66,"errors":false,"items":[{"index":{"_index":"my-index-filebeat","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":3,"status":201}}]}

output.elasticsearch:
#hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
hosts: 'https://XXXXXX.com:8443'
index: 'my-index-filebeat'
username: "XXXXXX"
password: "YYYYYY"
setup.template:
name: 'my-index-filebeat'
overwrite: true
enabled: false
pattern: 'my-index-filebeat*'
setup.ilm.enabled: false

It seems that enabled: false did the trick.

Thanks for your support

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.