Make stored documents untouchable - compliance reasons

Hello Team,

We use our Elasticsearch for a centralised logs storage. Due to compliance and security reasons we will need to harden it so the documents once written cannot be deleted/updated anymore from the same environment. The servers based in aws send:

  • syslog, nginx etc. entries using td-agent and
  • java applications use a logback plugin,
  • our developers send their java logs from their laptops plugged into our office network.

First I got surprised that I couldn't find anything about it after a day of searching but I understand Elasticsearch is supposed to work in a trusted environment.

Could you guys provide some ideas how we could approach it? My first thought was to use nginx as a reverse proxy with some rules to access/deny the requests but when I started with the list of allowed/denied methods and URLs I eventually gave up with the _bulk as I would need to force nginx to filter the traffic based on json content with some regexp patterns:

POST /{index}/{type}/
PUT /{index}/{type}/{id}?op_type=create
PUT /{index}/{type}/{id}/_create

~~~~~ What requests are not allowed?
PUT /{index}/{type}/{id}
POST /{index}/{type}/{id}/_update
POST /_bulk
POST /{index}/_bulk
POST /{index}/{type}/_bulk
  { "delete": { "_index": {index}, "_type": {type}, "_id": {id} }}
  { "create": { "_index": {index}, "_type": {type}, "_id": {id} }}
  { "update": { "_index": {index}, "_type": {type}, "_id": {id} }}

Our next approach was at least to check if any of the records were modified by searching for **_version** being greater than 1 but as that metadata is not indexed so it's not searchable and it would be flawed anyway.

I was also searching for 'immutable documents' or 'disable destructive operations' but those have different meanings here.

Am I missing any simple solutions? I think I will now focus on Elasticsearch itself looking for some fancy options or will take a look at **Shield** plugin what it can do. I don't know too much about Logstash, should I take a look at it?

Any advice appreciated.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.