Gelf to multiline filter messes up newlines

Elastic Stack Logstash
1

I'm trying to get the output of python stacktraces to show up nicely in Kibana. I am using the docker gelf log driver, and I am able to get the messages through logstash into ES. With the following config, tracebacks are correctly kept together, however there is something wrong with the newline processing.

Config:

input {
  gelf {
    type => docker
    port => 12201
  }
}

filter {
  multiline {
    pattern => "^\["
    negate => "true"
    what => "previous"
  }
  grok { match => { "container_name" => "^(?<compose_project>[a-z]+)_(?<service>.+)_\d+$" } }
  # mutate {
  #    replace => { "message" => "%{short_message}" }
  # }
}

output {
  elasticsearch { hosts => ["elasticsearch:9200"] }
  stdout { codec => rubydebug }
}

The result is the following in the rubydebug output:

{
            "version" => "1.1",
               "host" => "default",
      "short_message" => [
        [ 0] "Performing system checks...",
        [ 1] "Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7f5373863048>",
        ... // TRUNCATED
        [61] "File \"/usr/local/lib/python3.4/site-packages/django/utils/six.py\", line 685, in reraise",
        [62] "raise value.with_traceback(tb)",
        [63] "django.db.utils.OperationalError: (2003, \"Can't connect to MySQL server on '172.17.0.1' (111)\")"
    ],
              "level" => 6,
           "facility" => "",
           "@version" => "1",
         "@timestamp" => "2016-01-17T06:26:47.000Z",
        "source_host" => "172.17.0.1",
            "message" => "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
            "command" => "python manage.py runserver 0.0.0.0:8000",
                ... // TRUNCATED
               "type" => "docker",
               "tags" => [
        [0] "multiline"
    ],
    "compose_project" => "compose",
            "service" => "web_run"
}

So short_message becomes a neat list, but message only contains the newlines! When I enable the commented-out replace in the config, message contains all lines in short_message, but in a single line.

Both result in an illegibly field in Kibana: http://cl.ly/0V2u3j200V46

How can I get this correctly formatted?

2

@tino could you fix this?

I am having a problem where by default gelf is making a single event without any newline, making it very hard to read.

Or my event is sending multiple lines, not sure yet.