Filter for content of specific field of second index

Hello,
I'm trying to setup a filter that creates unique fingerprints if username does not already exists in index identities. Otherwise look up the value of the created pseudoId and put this in the event index. How do access the content of a specific field of a second index.
The code I'm using comes from the pseudonymization tutorial: https://www.elastic.co/de/blog/gdpr-personal-data-pseudonymization-part-1

    filter {

        ruby {

            code => "event.set('identities',[])"

        }

        # pseudonymise ip field

            #fingerprint ip

            fingerprint {

                method => "UUID" # method can be Sha128, SHA256, UUID ...

                source => ["ip"]

                key => "${FINGERPRINT_KEY}"

            }

            #create sub document under identities field

            mutate { add_field => { '[identities][0][key]' => "%{fingerprint}"  '[identities][0][value]' => "%{ip}" }  }

            #overwrite ip field with fingerprint

            mutate { replace => { "ip" => "%{fingerprint}" } }

        # pseudonymise username field

            #fingerprint username

            if "%{username}" not in [identities][value] {

            

                fingerprint {

                    method => "UUID"

                    source => ["username"]

                    key => "${FINGERPRINT_KEY}"

                }

            

                #create sub document under identities field

                mutate { add_field => { '[identities][1][key]' => "%{fingerprint}"  '[identities][1][value]' => "%{username}" } }

                #overwrite username field with fingerprint

                mutate { replace => { "username" => "%{fingerprint}" } }

                    

                #extract sub documents and yield a new document for each one into the LS pipeline. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#_inline_ruby_code

                ruby {

                    code => "event.get('identities').each { |p| e=LogStash::Event.new(p); e.tag('identities'); new_event_block.call(e); } "

                }

                #remove fields on original doc

                mutate { remove_field => ["fingerprint","identities"] add_field => { "source" => "fingerprint_pipeline" } }

            } 

    }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.