Merge extendedStackTrace received from log4j to regular stack trace

I am trying integrate our product with ELK. I use log4j and socket appender with JsonLayout and I see all my logs in Kibana. But stacktraces I get look like:

"thrown": {
    "extendedStackTrace": [
      {
        "file": "ApacheHttpClient4Handler.java",
        "method": "handle",
        "line": 187,
        "exact": false,
        "location": "jersey-apache-client4-1.19.1.jar",
        "class": "com.sun.jersey.client.apache4.ApacheHttpClient4Handler",
        "version": "1.19.1"
      },
      {
        "file": "EurekaIdentityHeaderFilter.java",
        "method": "handle",
        "line": 27,
        "exact": false,
        "location": "eureka-client-1.9.3.jar",
        "class": "com.netflix.discovery.EurekaIdentityHeaderFilter",
        "version": "1.9.3"
      }
    ],
    "localizedMessage": "java.net.ConnectException: Connection refused: connect",
    "name": "com.sun.jersey.api.client.ClientHandlerException",
    "cause": {
      "name": "java.net.ConnectException",
      "extendedStackTrace": [
        {
          "file": "DualStackPlainSocketImpl.java",
          "method": "waitForConnect",
          "line": -2,
          "exact": false,
          "location": "?",
          "class": "java.net.DualStackPlainSocketImpl",
          "version": "1.8.0_121"
        },
        {
          "file": "ApacheHttpClient4Handler.java",
          "method": "handle",
          "line": 173,
          "exact": false,
          "location": "jersey-apache-client4-1.19.1.jar",
          "class": "com.sun.jersey.client.apache4.ApacheHttpClient4Handler",
          "version": "1.19.1"
        }
      ],
      "commonElementCount": 29,
      "localizedMessage": "Connection refused: connect",
      "message": "Connection refused: connect"
    },
    "commonElementCount": 0,
    "message": "java.net.ConnectException: Connection refused: connect"
  }

I'm not very familiar with logstash but I'd like to see stacktraces as I'm used to. I took a look on many examples where "grok", "mutate", "json" filter plugins were used but I don't see the exapmle how to merge It the way It looks like a regular stack trace placed to separate tag.

Please advise something to me

I would like to see it like, not exactly but It should be just regular stack trace:

2018-09-04 20:18:37 com.netflix.discovery.DiscoveryClient [ERROR] DiscoveryClient_BOOK-GATEWAY-SERVICE/RUENSUKHOAD2C.was unable to refresh its cache! status = Cannot execute request on any known server com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112) ~[eureka-client-1.9.3.jar:1.9.3] at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134) ~[eureka-client-1.9.3.jar:1.9.3] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_121] at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[?:1.8.0_121] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.6.jar:4.5.6] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6] at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173) ~[jersey-apache-client4-1.19.1.jar:1.19.1] ... 29 more

Now My logstash.cong looks like that:

input {
  tcp {
    port => 9600
    codec => json
  }
}

filter {
  if [thrown] {
    ruby {
        code => "
            event.get('thrown')['extendedStackTrace'].map! { |v| v.map { |k, v| "#{v}" }.join(", ") }
        "
     }
  }
}

output {
  elasticsearch { hosts => ["elasticsearch-service:9200"] }
}

And It doesn't work

I've written a quick ruby filter for formatting an extended stack trace into the typical stringified version that should get you started:

def register(params)
  params = params.dup

  @source = params.delete('source') || fail(LogStash::ConfigurationError, 'no `source` provided!')
  @target = params.delete('target') || fail(LogStash::ConfigurationError, 'no `target` provided!')

  @joiner = params.delete('joiner') || "\n"
  @indent = params.delete('indent') || "\t"

  params.empty? || fail(LogStash::ConfigurationError, "unknown params #{params.inspect}")
end

def filter(event)
  thrown = event.get(@source)
  return if thrown.nil?

  event.set(@target, stacktrace_format(thrown))
rescue => e
  logger.warn("Exception while formatting extended stacktrace", :exception => e.message, :backtrace => e.backtrace)
  event.tag('_stacktraceformaterror')
end

def stacktrace_format(thrown)
  message = []
  message << thrown.fetch('localizedMessage') { thrown.fetch('message', '?')}
  thrown['extendedStackTrace'].each do |entry|
    message << "#{@indent}at #{entry.fetch('class','?')}.#{entry.fetch('method','?')}(#{entry.fetch('file','?')}:#{entry.fetch('line','?')}) ~[#{entry.fetch('location','?')}:#{entry.fetch('version','?')}]"
  end
  if thrown.include?('cause')
    message << 'Caused by:'
    message << stacktrace_format(thrown['cause'])
  end
  message.join(@joiner)
end

USAGE:

filter {
  ruby {
    path => "./extended-stack-trace-formatter.rb"
    script_params => {
      "source" => "[thrown]"
      "target" => "[backtrace]"
    }
  }
}

EXAMPLE OUTPUT:

{
          "host" => "castrovel.local",
     "backtrace" => "java.net.ConnectException: Connection refused: connect\n\tat com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar:1.19.1]\n\tat com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.9.3.jar:1.9.3]\nCaused by:\nConnection refused: connect\n\tat java.net.DualStackPlainSocketImpl.waitForConnect(DualStackPlainSocketImpl.java:-2) ~[?:1.8.0_121]\n\tat com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173) ~[jersey-apache-client4-1.19.1.jar:1.19.1]",
      "@version" => "1",
        "thrown" => {
                   "message" => "java.net.ConnectException: Connection refused: connect",
                     "cause" => {
                       "message" => "Connection refused: connect",
            "commonElementCount" => 29,
            "extendedStackTrace" => [
                [0] {
                        "line" => -2,
                    "location" => "?",
                       "class" => "java.net.DualStackPlainSocketImpl",
                     "version" => "1.8.0_121",
                      "method" => "waitForConnect",
                       "exact" => false,
                        "file" => "DualStackPlainSocketImpl.java"
                },
                [1] {
                        "line" => 173,
                    "location" => "jersey-apache-client4-1.19.1.jar",
                       "class" => "com.sun.jersey.client.apache4.ApacheHttpClient4Handler",
                     "version" => "1.19.1",
                      "method" => "handle",
                       "exact" => false,
                        "file" => "ApacheHttpClient4Handler.java"
                }
            ],
                          "name" => "java.net.ConnectException",
              "localizedMessage" => "Connection refused: connect"
        },
                      "name" => "com.sun.jersey.api.client.ClientHandlerException",
          "localizedMessage" => "java.net.ConnectException: Connection refused: connect",
        "commonElementCount" => 0,
        "extendedStackTrace" => [
            [0] {
                    "line" => 187,
                "location" => "jersey-apache-client4-1.19.1.jar",
                   "class" => "com.sun.jersey.client.apache4.ApacheHttpClient4Handler",
                 "version" => "1.19.1",
                  "method" => "handle",
                   "exact" => false,
                    "file" => "ApacheHttpClient4Handler.java"
            },
            [1] {
                    "line" => 27,
                "location" => "eureka-client-1.9.3.jar",
                   "class" => "com.netflix.discovery.EurekaIdentityHeaderFilter",
                 "version" => "1.9.3",
                  "method" => "handle",
                   "exact" => false,
                    "file" => "EurekaIdentityHeaderFilter.java"
            }
        ]
    },
          "tags" => [
        [0] "_rubyexception"
    ],
    "@timestamp" => 2018-09-05T18:30:25.552Z
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.