Hello Team,
We use our Elasticsearch for a centralised logs storage. Due to compliance and security reasons we will need to harden it so the documents once written cannot be deleted/updated anymore from the same environment. The servers based in aws send:
- syslog, nginx etc. entries using td-agent and
- java applications use a logback plugin,
- our developers send their java logs from their laptops plugged into our office network.
First I got surprised that I couldn't find anything about it after a day of searching but I understand Elasticsearch is supposed to work in a trusted environment.
Could you guys provide some ideas how we could approach it? My first thought was to use nginx as a reverse proxy with some rules to access/deny the requests but when I started with the list of allowed/denied methods and URLs I eventually gave up with the _bulk as I would need to force nginx to filter the traffic based on json content with some regexp patterns:
GET
POST /{index}/{type}/
HEAD
PUT /{index}/{type}/{id}?op_type=create
PUT /{index}/{type}/{id}/_create
~~~~~ What requests are not allowed?
PUT /{index}/{type}/{id}
DELETE
POST /{index}/{type}/{id}/_update
POST /_bulk
POST /{index}/_bulk
POST /{index}/{type}/_bulk
{ "delete": { "_index": {index}, "_type": {type}, "_id": {id} }}
{ "create": { "_index": {index}, "_type": {type}, "_id": {id} }}
{ "update": { "_index": {index}, "_type": {type}, "_id": {id} }}
Our next approach was at least to check if any of the records were modified by searching for **_version** being greater than 1 but as that metadata is not indexed so it's not searchable and it would be flawed anyway.
I was also searching for 'immutable documents' or 'disable destructive operations' but those have different meanings here.
Am I missing any simple solutions? I think I will now focus on Elasticsearch itself looking for some fancy options or will take a look at **Shield** plugin what it can do. I don't know too much about Logstash, should I take a look at it?
Any advice appreciated.
Thanks,