Kibana logs shows HTTP 403 (Forbidden)

Elastic Stack Beats
1

Hi,

We are using kibana for monitoring where we defined the a custom files for url and TCP monitoring in heartbeat with all details a month back.Since last week URL and TCP monitoring data shows "No results found" in kibana dashboard and from logs noticed HTTP 403 Forbidden error.
Please help me to fix the issue and snippit of log mentioned below
2019-05-29T11:57:23.195+0800 WARN elasticsearch/client.go:523 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf339ea88c7166a1, ext:1715053200758, loc:(*time.Location)(0x1fe1a00)}, Meta:common.MapStr(nil), Fields:common.MapStr{"tcp":common.MapStr{"rtt":common.MapStr{"connect":common.MapStr{"us":276}}, "port":0x20fb}, "monitor":common.MapStr{"ip":"X.X.X.X", "status":"up", "duration":common.MapStr{"us":349}, "host":"X.X.X.X", "scheme":"tcp"

2

A 403 error indicates a lack of permissions. It usually means a user/password combo are incorrect. Are you sure you've included the correct error message below? There is no mention of 403 in it.

3

Hi Andrew,

Below is the snippet from kibana.stdout log
{"type":"log","@timestamp":"2019-06-04T02:33:20Z","tags":["error","task_manager"],"pid":17644,"message":"Failed to poll for work: [cluster
_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana_task_manager/_doc/Maps-maps_tel
emetry/_update","query":{"if_seq_no":40,"if_primary_term":4,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\
",\"task\":{\"taskType\":\"maps_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{}}\",\"params
\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2019-05-27T04:27:32.931Z\",\"runAt\":\"2019-06-04T02:34:20.887Z\",\
"status\":\"running\"},\"kibana\":{\"uuid\":\"979cbc12-fc31-443f-9583-0071fb272f4b\",\"version\":6070299,\"apiVers
ion\":1}}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"rea
son\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason
\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}
{"type":"log","@timestamp":"2019-06-04T02:33:23Z","tags":["error","task_manager"],"pid":17644,"message":"Failed to poll for work: [cluster
_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana_task_manager/_doc/Maps-maps_tel
emetry/_update","query":{"if_seq_no":40,"if_primary_term":4,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\
",\"task\":{\"taskType\":\"maps_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{}}\",\"params
\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2019-05-27T04:27:32.931Z\",\"runAt\":\"2019-06-04T02:34:23.911Z\",\
"status\":\"running\"},\"kibana\":{\"uuid\":\"979cbc12-fc31-443f-9583-0071fb272f4b\",\"version\":6070299,\"apiVers
ion\":1}}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"rea
son\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason
\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}

4

Hi Andrew,

Copied entire error message from logs have shown below. please help me on this.
still we are seeing this error

5

It looks like you have not granted permissions for heartbeat to write to Elasticsearch. That would be the root cause.

6

Hi Andrew,

Can you help me on how to provide permissions for heartbeat to write to elasticsearch

7

Hi Andrew,

Can i have an update on this

Thanks

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.