Logstash filter cipher - static_iv instead of random iv does NOT WORK

Elastic Stack Logstash
1

Dear community member,

I use logstash-filter-cipher to encrypt my "message" field; the confituration works pretty well the "message" is well crypt BUT the encryption is base on "iv_random" therefore the same "string" is encrypt differently each time.

my goal is to have the same string hash the same way; therefore I tryed with a "static iv" instead of a "dynamic (random) iv"; BUT it's not working :confused:

Any idea, any example with a different algorythmn, any advices for me so far?

Thanks in advance for you help

kr,
O.

logstash version: 6.2.0
logstash cipher plugin version: version: v3.0.1 - Released on: 2017-11-07
logstash-filter-cipher Official guide: Cipher filter plugin | Logstash Reference [8.11] | Elastic

Assuming I have the following logstash cipher configuration file:

input{
	stdin{}
}

filter{
	# add cypher to encrypt message
	# replace message by encrypt message
	cipher {
		algorithm => "aes-256-cbc"
		cipher_padding => 1

		# Use a static "iv"
		#iv => "1234567890123456"

		# OR use a random IV per encryption
		iv_random_length => 16

		key => "12345678901234567890123456789012"
		key_size => 32

		mode => "encrypt"
		source => "message"
		target => "message_crypted"
		base64 => true

		# the maximum number of times the
		# internal cipher object instance
		# should be re-used before creating
		# a new one, default to 1
		#
		# On high volume systems bump this up
		max_cipher_reuse => 1
	}
	mutate {
		replace => { "message" => "%{[message_crypted]}" }
		remove_field => ["message_crypted", "@timestamp", "host", "@version"]
	}
}

output{
  stdout { codec => rubydebug }
}

In the following I lanched the same logstash config for the same string "message" and have a differente result with a random iv

$ echo "super_message_string" | logstash --verbose -f cipher_encrypt.rb && sleep 5 ; echo "super_message_string" | logstash --verbose -f cipher_encrypt.rb
Sending Logstash's logs to /SandBox/elastic/es-620/logstash-6.2.0/logs which is now configured via log4j2.properties
[2018-03-29T17:51:37,941][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/netflow/configuration"}
[2018-03-29T17:51:37,973][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/fb_apache/configuration"}
[2018-03-29T17:51:38,306][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-29T17:51:39,192][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.0"}
[2018-03-29T17:51:39,774][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-29T17:51:45,555][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-29T17:51:45,783][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7f496feb run>"}
[2018-03-29T17:51:46,029][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
    "message" => "94pF+RNHQGCiv1kQXPwUidQAXDo4fYbp0rBMAHLkgP04CuHge3t2fYKjLXSYykGj"
}
[2018-03-29T17:51:46,812][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x7f496feb run>"}
Sending Logstash's logs to /SandBox/elastic/es-620/logstash-6.2.0/logs which is now configured via log4j2.properties
[2018-03-29T17:52:16,555][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/netflow/configuration"}
[2018-03-29T17:52:16,578][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/SandBox/elastic/es-620/logstash-6.2.0/modules/fb_apache/configuration"}
[2018-03-29T17:52:16,832][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-29T17:52:17,591][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.0"}
[2018-03-29T17:52:18,143][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-29T17:52:24,017][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-29T17:52:24,223][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7b546ed6 run>"}
[2018-03-29T17:52:24,432][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
{
    "message" => "VwgdEchuFShSuMqXvev3KNsqYwUl9Gao+K/8s4GEJ1kMkIEMAd4nRtoozFasFRFq"
}
[2018-03-29T17:52:25,354][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x7b546ed6 run>"}

as you can see the hash of message is defferent ;

if I use the static iv I received the following error message:

[2018-03-29T18:00:07,193][ERROR][logstash.filters.cipher  ] Missing a required setting for the cipher filter plugin:

  filter {
    cipher {
      iv_random_length => # SETTING MISSING
      ...
    }
  }

In the documentation the iv_random_length is well a type: number BUT it is not "required" :confused:

2

Then you are not looking for encryption; what is your goal?

The static iv setting was deprecated and is now obsolete because using a static initialisation vector is a known bad practice that renders useless many of the desired properties of encryption; see also: Crypto Fails — Crypto Noobs #1: Initialization Vectors

3

Hi Yaauie, thank you for your quick reply; my goal is quiet simple:

Imagine you have a data set with a "user" field name, or even a "bank_account" field" or whatever in you data set which for securtiy reason, you want these field to be encrypt.

:warning: BUT in the same time you don't want to make the life of your data analyst difficult.

:information_source: Therefore in order to let them query, aggregate, sort your data set you encrypt the same field "user" string value the same way.

Each time you script a user_file:"orsius" it must be encrypted "xxxxyyyyy" .

At the end you data analyst team could give you the top 10 users without knowing their real name. And the team with the proper rights will be able to decrypt the information (reverse the encryption process) and know the real name.

I hope my final goeal appears more cleary to you now?

:pushpin: To reach this goal, I want to keep my data indexing process as simple as possible by simply used elastic component and no other application or additional script.

5

I see where you're coming from, and although I do not specialise in security, I do know that a static IV reduces what could otherwise be a secure storage mechanism to mere obfuscation that is very weak against correlation attacks (that is: an abuser who has the encrypted ciphertext from many plaintexts can crack the encryption key with significantly less computational effort); it is a known anti-pattern for a reason.

In this type of situation, I would recommend storing a deterministically-generated foreign key in Elasticsearch (e.g., a sha-sum of ${SALT}+${VALUE}, where SALT is a consistent long sequence of bytes, and VALUE is the value being obfuscated), and either store the sensitive data in a separately-secured system or, if using X-Pack Security, on the same document in a secured field for which your analyst users do not have permission in Elasticsearch.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.