Fingerprint plugin crashes

Elastic Stack Logstash
1

Hello there.
I'm using ELK 5.1.1 stack and right now was trying to configure fingerprint plugin to generate hash-code for URL links.
Here's part of configuration:

input {
      http{
            type => "frontend_event"
      }
}
filter {
    de_dot{

    }
    if [type] == 'frontend_event' {
        date {
            match => ["timestamp", "ISO8601", "UNIX_MS"]
            remove_field => ["timestamp"]
        }
        mutate{
            add_field => {
            "user_agent_unparsed" => "%{[headers][http_user_agent]}"
            }
        }
        useragent {
            source => "user_agent_unparsed"
            target => "user_agent"
            remove_field => ["user_agent_unparsed"]
        }
        if [eventType] == 'SEARCH' {
            fingerprint {
                source => ["url"]
                target => "search_url_hash"
                method => "SHA1"
            }
        }
        if [eventType] == 'OPEN_PAGE' {
            fingerprint {
                source => ["searchUrl"]
                target => "search_url_hash"
                method => "SHA1"
            }
        }
    }
}
output{
    elasticsearch{}
}

And while logstash is setting up, I receive following error:

15:25:55.325 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: translation missing: en.logstash.agent.configuration.invalid_plugin_register>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fingerprint-3.0.2/lib/logstash/filters/fingerprint.rb:60:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:230:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:230:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:183:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:292:in `start_pipeline'"]}

From my point of view, it's kind of a bug, but maybe someone could find any config-related problems, I do appreciate any help.

P.S. Also logstash -f /etc/logstash/conf.d/ -t returns:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ -t
Could not find log4j2 configuration at path /etc/logstash/log4j2.properties. Using default config which logs to console
Configuration OK
15:25:10.371 [LogStash::Runner] INFO  logstash.runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
2

The poor error message is a bug, but the root cause in an incomplete configuration. This is the error message that should've been presented to you (according to the stack trace):

Key value is empty. please fill in a subnet prefix length

What's confusing here is that this message only comes when using IPV4_METHOD as the method, which you're not according to the configuration you posted. Anyway, the root cause is that you're not setting the key configuration option.

1 Like
3

I do confirm, that adding key fixes error, but, probably documentation for this plugin should be adjusted in a following way:

  1. As a raw Hash functions do not require any keys, probably smth like HMAC-Hash functions are used, that's why Key is required even for SHA1 - you need to point, that not SHA1, SHA-256, etc. are used, but HMAC-SHA1, HMAC-SHA-256 etc.

  2. Key should be required, while you cannot use this plugin without key.

Also, I've looked through plugin github repo and found that there is already an issue about HMAC and simple hash function: Provide hash functions in addition to HMAC functions #18

I do think, that at least documentation should be updated, if implementation update cannot be done easily.

Anyway, thank you for help, @magnusbaeck.

1 Like
4
  1. Key should be required, while you cannot use this plugin without key.

That's not quite true. The MURMUR3, UUID, and PUNCTUATION methods don't make use of the key.

5

OK, maybe you're right as for those 3 algorithms, but anyway documentation should be somehow adjusted, cause right now it's not clear that you should set some key for Hash-function (really, hash-functions do NOT require any key, but HMAC do require).

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.