Dear community member,
I use logstash-filter-cipher to encrypt my "message" field; the confituration works pretty well the "message" is well crypt BUT the encryption is base on "iv_random" therefore the same "string" is encrypt differently each time.
my goal is to have the same string hash the same way; therefore I tryed with a "static iv" instead of a "dynamic (random) iv"; BUT it's not working 
Any idea, any example with a different algorythmn, any advices for me so far?
Thanks in advance for you help
kr,
O.
logstash version: 6.2.0
logstash cipher plugin version: version: v3.0.1 - Released on: 2017-11-07
logstash-filter-cipher Official guide: Cipher filter plugin | Logstash Reference [8.11] | Elastic
Assuming I have the following logstash cipher configuration file:
input{
stdin{}
}
filter{
# add cypher to encrypt message
# replace message by encrypt message
cipher {
algorithm => "aes-256-cbc"
cipher_padding => 1
# Use a static "iv"
#iv => "1234567890123456"
# OR use a random IV per encryption
iv_random_length => 16
key => "12345678901234567890123456789012"
key_size => 32
mode => "encrypt"
source => "message"
target => "message_crypted"
base64 => true
# the maximum number of times the
# internal cipher object instance
# should be re-used before creating
# a new one, default to 1
#
# On high volume systems bump this up
max_cipher_reuse => 1
}
mutate {
replace => { "message" => "%{[message_crypted]}" }
remove_field => ["message_crypted", "@timestamp", "host", "@version"]
}
}
output{
stdout { codec => rubydebug }
}
In the following I lanched the same logstash config for the same string "message" and have a differente result with a random iv
$ echo "super_message_string" | logstash --verbose -f cipher_encrypt.rb && sleep 5 ; echo "super_message_string" | logstash --verbose -f cipher_encrypt.rb
Sending Logstash's logs to /SandBox/elastic/es-620/logstash-6.2.0/logs which is now configured via log4j2.properties
[2018-03-29T17:51:37,941][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/netflow/configuration"}
[2018-03-29T17:51:37,973][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/fb_apache/configuration"}
[2018-03-29T17:51:38,306][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-29T17:51:39,192][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.0"}
[2018-03-29T17:51:39,774][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-29T17:51:45,555][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-29T17:51:45,783][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7f496feb run>"}
[2018-03-29T17:51:46,029][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
"message" => "94pF+RNHQGCiv1kQXPwUidQAXDo4fYbp0rBMAHLkgP04CuHge3t2fYKjLXSYykGj"
}
[2018-03-29T17:51:46,812][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x7f496feb run>"}
Sending Logstash's logs to /SandBox/elastic/es-620/logstash-6.2.0/logs which is now configured via log4j2.properties
[2018-03-29T17:52:16,555][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/netflow/configuration"}
[2018-03-29T17:52:16,578][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/fb_apache/configuration"}
[2018-03-29T17:52:16,832][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-29T17:52:17,591][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.0"}
[2018-03-29T17:52:18,143][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-29T17:52:24,017][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-29T17:52:24,223][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7b546ed6 run>"}
[2018-03-29T17:52:24,432][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
"message" => "VwgdEchuFShSuMqXvev3KNsqYwUl9Gao+K/8s4GEJ1kMkIEMAd4nRtoozFasFRFq"
}
[2018-03-29T17:52:25,354][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x7b546ed6 run>"}
as you can see the hash of message is defferent ;
if I use the static iv I received the following error message:
[2018-03-29T18:00:07,193][ERROR][logstash.filters.cipher ] Missing a required setting for the cipher filter plugin:
filter {
cipher {
iv_random_length => # SETTING MISSING
...
}
}
In the documentation the iv_random_length is well a type: number BUT it is not "required"
BUT in the same time you don't want to make the life of your data analyst difficult.
Therefore in order to let them query, aggregate, sort your data set you encrypt the same field "user" string value the same way.
To reach this goal, I want to keep my data indexing process as simple as possible by simply used elastic component and no other application or additional script.