Hello,
I'm trying to setup a filter that creates unique fingerprints if username does not already exists in index identities. Otherwise look up the value of the created pseudoId and put this in the event index. How do access the content of a specific field of a second index.
The code I'm using comes from the pseudonymization tutorial: https://www.elastic.co/de/blog/gdpr-personal-data-pseudonymization-part-1
filter {
ruby {
code => "event.set('identities',[])"
}
# pseudonymise ip field
#fingerprint ip
fingerprint {
method => "UUID" # method can be Sha128, SHA256, UUID ...
source => ["ip"]
key => "${FINGERPRINT_KEY}"
}
#create sub document under identities field
mutate { add_field => { '[identities][0][key]' => "%{fingerprint}" '[identities][0][value]' => "%{ip}" } }
#overwrite ip field with fingerprint
mutate { replace => { "ip" => "%{fingerprint}" } }
# pseudonymise username field
#fingerprint username
if "%{username}" not in [identities][value] {
fingerprint {
method => "UUID"
source => ["username"]
key => "${FINGERPRINT_KEY}"
}
#create sub document under identities field
mutate { add_field => { '[identities][1][key]' => "%{fingerprint}" '[identities][1][value]' => "%{username}" } }
#overwrite username field with fingerprint
mutate { replace => { "username" => "%{fingerprint}" } }
#extract sub documents and yield a new document for each one into the LS pipeline. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#_inline_ruby_code
ruby {
code => "event.get('identities').each { |p| e=LogStash::Event.new(p); e.tag('identities'); new_event_block.call(e); } "
}
#remove fields on original doc
mutate { remove_field => ["fingerprint","identities"] add_field => { "source" => "fingerprint_pipeline" } }
}
}