Cannot index event publisher.Event (elasticsearch/client.go:502)

[shaharyar]

I set up APM in my Nodejs Express app. It starts fine, creates a new index in elasticsearch. When API return errors it only logs first few errors in elasticsearch (3,4 or 7 sometimes, its random) and then on starts throwing following error:

2018-04-10T19:22:28.516+0500 DEBUG [elasticsearch] elasticsearch/client.go:303 PublishEvents: 2 events have been published to elasticsearch in 88.166183ms.
2018-04-10T19:22:28.516+0500 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x16e3600, ext:63658966937, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"context":common.MapStr{"service":common.MapStr{"name":"api", "agent":common.MapStr{"name":"nodejs", "version":"1.4.0"}, "language":common.MapStr{"name":"javascript"}, "runtime":common.MapStr{"name":"node", "version":"v8.11.1"}}, "system":common.MapStr{"hostname":"shaharyar-g50", "architecture":"x64", "platform":"linux"}, "process":common.MapStr{"pid":31964, "title":"node", "argv":string{"/usr/bin/node", "/home/shaharyar/Workspace/Github/API-NodeJS/app/server.js"}}}, "processor":common.MapStr{"name":"transaction", "event":"transaction"}, "transaction":common.MapStr{"id":"96394f11-2bd3-42cd-ae79-3520332ab6d1", "name":"GET /users/:username", "duration":common.MapStr{"us":838132}, "type":"request", "result":"success", "sampled":false}, "beat":common.MapStr{"name":"shaharyar-g50", "hostname":"shaharyar-g50", "version":"6.2.3"}}, Private:interface {}(nil)}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2018-04-10T19:22:28.516+0500 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x14163640, ext:63658966941, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"transaction":common.MapStr{"result":"success", "sampled":false, "id":"e9521401-d143-406e-80e8-7b8cab4c205a", "name":"GET /users/:username", "duration":common.MapStr{"us":351801}, "type":"request"}, "context":common.MapStr{"system":common.MapStr{"architecture":"x64", "platform":"linux", "hostname":"shaharyar-g50"}, "process":common.MapStr{"pid":31964, "title":"node", "argv":string{"/usr/bin/node", "/home/shaharyar/Workspace/Github/API-NodeJS/app/server.js"}}, "service":common.MapStr{"agent":common.MapStr{"name":"nodejs", "version":"1.4.0"}, "language":common.MapStr{"name":"javascript"}, "runtime":common.MapStr{"version":"v8.11.1", "name":"node"}, "name":"api"}}, "processor":common.MapStr{"name":"transaction", "event":"transaction"}, "beat":common.MapStr{"name":"shaharyar-g50", "hostname":"shaharyar-g50", "version":"6.2.3"}}, Private:interface {}(nil)}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

I am throwing same errors as those who got indexed successfully, but still it throws the same above error.

When I delete the index from elasticsearch, and throws new logs. It creates new index, stores few logs and then suddenly starts throwing above error. I have tried it many times (deleting the index).

[roncohen]

Hi!

This looks like a problem with your Elasticsearch setup.
Try the solution posted here: FORBIDDEN/12/index read-only / allow delete (api)]

Does that help?

/Ron

[shaharyar]

The same error came while setting up kibana, and I ran the query mentioned in your provided link and error got resolved.

Now same error is coming with APM indices. Why is it coming over and over again? Will I have to run this query everytime I create a new index?

[jasontedor]

Are you low on disk space? This looks potentially like what would happen if you are teetering in the edge of the flood stage disk watermark and we flip indices to read only (to avoid completely filling up).

[shaharyar]

Yes, the low storage was the problem.
Thanks

[jasontedor]

You’re welcome.