Fingerprinting stack traces

Elastic Stack Logstash
1

Hi all,

could I get some advice as to how I can efficiently fingerprint stack traces using Logstash?

I am aware of the fingerprint filter, but it's not quite enough. I also want to ignore everything in the message that is before the stack trace, since the error message may contain data that changes. For example: if the stack trace is triggered by a primary key violation and the key values are logged, then the hash of the message would differ, but I want to consider the error still the same.

What it basically comes down to is a rule that says "ignore everything before the first line matching a prefix", with that prefix being akin to "\tat" - it may vary a bit by application, but we can manage that on a per-application basis.

Is there a way to do that efficiently in Logstash without a custom plugin?

In the long run we may need to filter out any line numbers present. We are trying to report on error trends across multiple releases, if the line numbers in a stack trace are present, then even adding a line in front of any involved code would change the fingerprint, which means we will have to collate additional fingerprints.

This second one is not much of an issue at the moment, the first application we are targeting is a .NET application that doesn't actually write line numbers. But we are looking at some Java applications that do log line numbers as well, which may require that second normalization.

Again: is there a way to do that efficiently without custom plugin?

Cheers,
Peter

2

If the messages are consistent enough, all you'd have to do is use the dissect filter or grok on it to separate the part you want to keep into its own field, and then use the fingerprint filter on that. The same thing applies to removing line numbers.

Even if they're able to be grouped consistently, you could still use grok or dissect to match them enough to break it down to make a field you want.

3

Thanks, Aaron.

I came up with this configuration:

if ( [message] =~ /(?m).*\n\s+at/ ) {
  grok {
    match => [ "message", "(?m)%{DATA}\n(?<stacktrace>\s+at.*)" ]
  }
  fingerprint {
    method => "SHA1"
    source => [ "stacktrace" ]
    target => "stacktrace_id"
  }
}

The grok part seems to work fine, but the fingerprint plugin fails with these errors in the log:

[2017-05-05T12:04:23,954][ERROR][logstash.pipeline        ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x713e24a @id=\"3a6701b48090e619946678a06eac772f4892e39c-11\", @klass=LogStash::Filters::Fingerprint, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x6f3f6b9f @metric=#<LogStash::Instrument::Metric:0x4c9ee531 @collector=#<LogStash::Instrument::Collector:0x4e6df34c @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x56745979 @store=#<Concurrent::Map:0x4283b3bb @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x798f61b7>, @fast_lookup=#<Concurrent::Map:0x58d91a59 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"3a6701b48090e619946678a06eac772f4892e39c-11\", :events]>, @logger=#<LogStash::Logging::Logger:0x73534ff8 @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x3e79490>>, @filter=<LogStash::Filters::Fingerprint method=>\"SHA1\", source=>[\"stacktrace\"], target=>\"stacktrace_id\", id=>\"3a6701b48090e619946678a06eac772f4892e39c-11\", enable_metric=>true, periodic_flush=>false, base64encode=>false, concatenate_sources=>false>>", :error=>"translation missing: en.logstash.agent.configuration.invalid_plugin_register"}
[2017-05-05T12:04:24,452][ERROR][logstash.agent           ] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: translation missing: en.logstash.agent.configuration.invalid_plugin_register>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fingerprint-3.0.2/lib/logstash/filters/fingerprint.rb:60:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:257:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:278:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:207:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `start_pipeline'"]}

I installed the plugin like this:

user@host:/usr/share/logstash$ sudo bin/logstash-plugin install logstash-filter-fingerprint

Logstash itself is coming from the APT repository.

Any idea what could be wrong?

Cheers,
Peter

4

I figured out that the error message was a red herring. The issue was that I did not specify the 'key' field, which is mandatory for the "SHA1" hashing.

Documentation and error reporting are a bit lacking here, although the code on github seems to handle the error, but I'm not seeing the corresponding message.

Anyway: it's working now.

Cheers,
Peter

5

Thanks for the feedback. If you have some ideas, we'd love for you to update the docs yourself with a pull request for https://github.com/logstash-plugins/logstash-filter-fingerprint/blob/master/docs/index.asciidoc

Glad to hear it's working for you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.