Logstash - obscure sensitive fields before putting them to the ES

Elastic Stack Logstash
#1

Hi,

Recently I came up with the next question.

Imagine the next, I have an application which writes the logs. Each log is simple JSON object with the depth of N. Developers don't think about the values they put it the logs, we can find password, IBAN, SSN fields there on any of levels.

All the logs are put to the Redis queue, then logstash reads from Redis and puts them to the ES.

Is there any option right now, where I can configure logstash in the way, that it will parse the JSON object, iterate over it and will obscure the value of the "sensitive" field (meaning change 5 symbols to *, for example)?

I have tried to search something in this direction, but without any success right now.

Any thoughts?

Regards,

#2

I don't think there's a stock plugin for this. You could do it with a ruby filter and write a custom filter.

#3

Thanks, that looks like a possible solution.

#4

What about https://www.elastic.co/guide/en/logstash/current/plugins-filters-anonymize.html

#5

What about https://www.elastic.co/guide/en/logstash/current/plugins-filters-anonymize.html

Ah, yes. If the name of the field(s) containing sensitive information is known then this is a good option.

#6

regarding this plugin, I have several questions:

  1. I haven't understood if this plugin checks only the first level of the log message for the fields with the given name, or it does it recursively?
  2. As I understand, this plugin would replace the content of the field to the hash. Which is ok and is a solution, however, it's different from obscure method.

Thanks for pointing me to this plugin.

Regards,

#7
  1. What do you mean by levels? Logs don't usually have levels.
  2. It's the same, you're replacing the data with something else. It's obscuring.
#8

The filter supports arrays of values but I don't think it supports nested fields, i.e. if you have

{
  "foo": {
    "bar": "baz"
  }
}

you can point if to [foo][bar] to obscure the "baz" string but you can't tell it to obscure [foo] and all subfields.

#9

Yes, I meant searching for the keys inside the JSON object.
Thanks

#10