Filebeat - CryptoJS is not defined

Elastic Stack Elasticsearch
1

Hello Team,
I was trying to implement the hashing using Filebeat script processor. But Filebeat doesnt have Crypto module by default and staring below error. Is there any option to achieve this functionality?

Geting below Javascript error.

ReferenceError: CryptoJS is not defined at inline.js:19:20(7)

The below mentioned code I have used in the Filebeat processors:

              var anonymizedMessage = message.replace(regex, function(match) {
                  var randomString = generateRandomString(8); // Generate a random salt
                  var hash = CryptoJS.SHA256(match + randomString).toString(CryptoJS.enc.Base64); // Generate hash using SHA-256
                  return hash;
              });

It would be more helpful if I get any suggestion or idea to resolve this?

Thanks
Siva

2

The script processor does not have any external dependencies as explained in the documentation.

The processor uses a pure Go implementation of ECMAScript 5.1 and has no external dependencies.

But what are you trying to achieve? Maybe it is possible using other processors on Filebeat or even something in an Ingest pipeline.

3

To achieve hashing in Filebeat without CryptoJS, consider preprocessing logs with an external script or using Logstash with a Ruby filter for hashing, as Filebeat doesn't support external JavaScript libraries like CryptoJS.

4

If you opt for the Logstash route, you can use the Ruby filter plugin like this:

rubyCopy code

filter {
  ruby {
    code => "
      require 'digest'
      event.get('message').gsub!(/your_regex_here/) do |match|
        random_string = SecureRandom.hex(4) # Generates a random string
        hash = Digest::SHA256.base64digest(match + random_string)
        hash
      end
    "
  }
}
5

Hi @siva0030,

@leandrojmp and @rtwolfe94022 are absolutely right, you can't use external dependencies in the script processor. It looks like you're trying to generate a hash, so I would recommend taking a look at the fingerprint processor to see if that will do what you need.

Hope that helps!

6

Hi @leandrojmp @rtwolfe94022 @carly.richmond,

Thank you for your time and response.
I apologize for not providing more context initially. My objective is to replace sensitive information in log messages with hashed values before sending them to Logstash. Specifically, I aim to use a script processor to generate unique hash values and replace occurrences of sensitive data such as "SECURITY***" with hashes, so used a below mentioned hashing algorithm like SHA-256 in combination with a random salt.

Code below.

processors:
  - script:
      lang: javascript
      id: generate_random_hashes
      source: |
        function generateRandomString(length) {
            // Generate a random string of specified length with uppercase letters
            var result = '';
            var characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
            var charactersLength = characters.length;
            for (var i = 0; i < length; i++) {
                result += characters.charAt(Math.floor(Math.random() * charactersLength));
            }
            return result;
        }          
        
        function process(event) {
            var message = event.Get("message");
            var regex = /(SECURITY\w+)|(SECURITY\d+)/g;
            
            // Replace each SECURITY keywords with a unique hash value
            var anonymizedMessage = message.replace(regex, function(match) {
                var randomString = generateRandomString(8); // Generate a random salt
                var hash = CryptoJS.SHA256(match + randomString).toString(CryptoJS.enc.Base64); // Generate hash using SHA-256
                return hash;
            });
            
            // Add a field indicating anonymization
            if (anonymizedMessage !== event.Get("message")) {
                event.Put("anonymized", true);
                event.Put("message", anonymizedMessage);
            }
            
            return event;
        }

Here, The generateRandomString function generates a random salt with the specified length.
Each occurrence of SECURITY in the message is replaced with a unique hash value generated by combining the SECURITY with the random salt and then hashing it using SHA-256.
The resulting hash value is added to the anonymizedMessage.
This hash value is unique for each occurrence of SECURITY keyword due to the random salt, ensuring that different occurrences are replaced with different hash values.

But It seems that Filebeat doen't have the CryptoJS library by default and due to that, its stating the below Javascript error.

ReferenceError: CryptoJS is not defined at inline.js:19:20(7)

@carly.richmond - as you mentioned, I could see there is an Algorithm (md5, sha1, sha256, sha384, sha512, xxhash) available in fingerprint processor.
I'm not sure how to utilize this processor for my usecase but anyway I will try it out.

Once again Thank You all!

7

Hi Team,
Good morning!

I attempted to utilize the Fingerprint processor to substitute a specific field with encoded values using the sha256 algorithm (method='sha256'). However, I encountered an issue where the Fingerprint processor is replacing the entire message instead of just the particular fields. Can someone assist me in achieving this goal? Below, I've provided the code I used and the input data. Please review and advise if I've misconfigured anything.

Input CSV data:

2024-02-29|16:20:20|100001|0.00|10.574|3|317876|sebes|0|0|0|DevOps01

Expecting output message as below:

2024-02-29|16:20:20|<hash/encoded value>|<hash/encoded value>|10.574|3|317876|sebes|0|0|0|DevOps01

But correctly what am getting in the message field is message" => "tE20NNHeIXw="

  processors:
    - dissect:
        tokenizer: "%{zero_column}|%{first_column}|%{second_column}|%{third_column}|%{remaining_columns}"
        field: "message"
        target_prefix: "dissected"
    - fingerprint:
        target_field: "message"
        method: "xxhash"
        salt: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
        fields: ["dissected_third_column", "dissected_second_column"]
        overwrite: true
        ignore_missing: true
        ignore_failure: true
        encoding: "base64"

thanks in advance!

8

You are telling it to put the hash in the message field. Change this to another field and see if that resolves the issue.

9

Hi @Christian_Dahlqvist - Thanks for your response!

I utilized different field and I was expecting each field should contain the hashed/encoded values. However, I encountered another issue where, when attempting to replace two different columns (fields), only one encoded value is present in the new field instead of two separate entries.

My intention is to replace those specific fields and update the same in the original message, as illustrated below

2024-02-29|16:20:20|<hash/encoded value>|<hash/encoded value>|10.574|3|317876|sebes|0|0|0|DevOps01
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.