Configuration using filebeat with static index name

Elastic Stack Beats
1

Hi,

I would like to know how I need to configure filebeat to use an existing index(it's empty). What's the configuration to do so? So far I tried in a helm chart:

 output.elasticsearch:
      hosts: 'https://XXXXXX.com:8443'
      index: 'my-index-filebeat'
      username: "XXXXX"
      password: "YYYYY"
    setup.template:
      name: 'my-index-filebeat'
      overwrite: true
      enabled: true
      pattern: 'my-index-filebeat'

I cannot create index in this elasticsearch server, only use this one.

Thanks in advance

2

It sounds like you are running into this issue: https://github.com/elastic/beats/issues/11866.

The workaround is to set setup.ilm.enabled: false in your Filebeat configuration. Could you try that and see if the my-index-filebeat template and my-index-filebeat index get created?

Thanks,

Shaunak

3

Thanks Shanuak,

I tried with that option but unfortunately it doesn't work.

My case it's a little bit uncommon. I need to use the my-index-filebeat index that was created by the elasticsearch admins in my company. I cannot create any index in the elasticsearch server.
What should I do to make it work?

Logs:

2019-10-17T13:56:00.894Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://XXXXXXXXXX.com:8443)) with 4 reconnect attempt(s)
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:191 done
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-10-17T13:56:00.894Z INFO [publisher] pipeline/retry.go:168 done
2019-10-17T13:56:00.903Z INFO elasticsearch/client.go:743 Attempting to connect to Elasticsearch version 6.8.2
2019-10-17T13:56:00.914Z INFO template/load.go:169 Existing template will be overwritten, as overwrite is enabled.
2019-10-17T13:56:00.984Z INFO template/load.go:108 Try loading template my-index-filebeat to Elasticsearch

4

Indexing into an existing index should be fine; I just tested it to make sure.

As a test, could you try to send a Bulk API request, using the same username+password as in your Filebeat configuration, to my-index-filebeat from the command line (using curl or equivalent) on the host where Filebeat is running?

That'll tell us if a) a connection can be made from the Filebeat host to the Elasticsearch cluster and b) if the configured user has sufficient privileges to bulk index documents into my-index-filebeat.

Also, are those all the logs you are seeing for Filebeat? Specifically, are there any log messages showing errors or warnings?

Finally, which version of Filebeat are you using?

Shaunak

5

As requested:

$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }

curl -s -H "Content-Type: application/x-ndjson" -XPOST https://XXXXXX.com:8443/_bulk --data-binary "@requests" -u XXXXXX; echo
Enter host password for user 'XXXXXX':
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: type is missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: type is missing;"},"status":400}

  • Example Connection:

$ curl -XGET "https://XXXXXXX.com:8443/my-index-filebeat/_search?pretty=true&q=*:*" -u XXXXXX
Enter host password for user 'XXXXXX':
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" :
}
}

  • I will double check the permissions with our team.

  • Only I have these INFO messages, get stuck in Try loading template ....

  • Filebeat version: 7.3.2

Many Thanks.

6

Can you try changing this line in requests to:

{ "index" : { "_index" : "test", "_type": "_doc", "_id" : "1" } }

and make the curl request again please?

7

{"took":9,"errors":true,"items":[{"index":{"_index":"test","_type":"_doc","_id":"1","status":403,"error":{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [XXXXX]"}}}]}

Just a reminder, I cannot create index only using the existing one.

Thanks.

8

So is test not an existing index? Can you try the bulk request from curl using the same existing index that you would use in your output.elasticsearch.index setting?

9

Sorry, I didn't changed the name of the index, this is the result with my index:

{"took":66,"errors":false,"items":[{"index":{"_index":"my-index-filebeat","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":3,"status":201}}]}

10

output.elasticsearch:
#hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
hosts: 'https://XXXXXX.com:8443'
index: 'my-index-filebeat'
username: "XXXXXX"
password: "YYYYYY"
setup.template:
name: 'my-index-filebeat'
overwrite: true
enabled: false
pattern: 'my-index-filebeat*'
setup.ilm.enabled: false

It seems that enabled: false did the trick.

Thanks for your support

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.