Logstash 5.1 refuses to "boot up" (wrong configuration)

[moonwhaler (Chris Pete)]

Hello there!

I am currently using the latest Debian (Jessie) packages from Logstash 5.1 (in conjunction with ES 5.1 and Kibana 5.1), but Logstash is driving me nuts. (Logstash 2.x was working fine, even testing the configuration worked well, something I cannot say for 5.1), anyway. I am currently fighting with this configuration:

input {
  beats {
    port => 5044
  }
}

filter {
    mutate {
      strip => "message"
    }

    grok {
      patterns_dir => [ "/etc/logstash/patterns" ]
      tag_on_failure => [ ]
      break_on_match => true

      match => {
        "message" => [
          "%{TIMESTAMP_ISO8601:logdate}[\t ]{1,}%{LOGLEVEL:loglevel}[\t ]{1,}\[%{DATA:thread}\][\t ]{1,}%{DATA:class}\(%{DATA:source_filename}:%{NUMBER:source_linenumber}\):[\t ]{1,}%{GREEDYDATA:message}",
          "%{TIMESTAMP_GERMAN:logdate}[\t ]{1,}%{LOGLEVEL:loglevel}:[\t ]{1,}%{GREEDYDATA}:[\t ]{1,}%{GREEDYDATA:message}"
        ]
      }

      overwrite => [ "message" ]
    }

    if [source] =~ /.+/ {
      grok {
        break_on_match => false
        tag_on_failure => [ ]

        match => [ "source", "(?<tool_complete_name>([A-Z]{2,3}-\d{1,3}))" ]
        match => [ "tool_complete_name", "(?<tool_id>([A-Z]{2,3}))" ]
        match => [ "tool_complete_name", "(?<tool_instance>(\d{1,3}))" ]

        remove_field => [ "tool_complete_name" ]
      }
    }

    date {
      match => [ "logdate", "yyyy-MM-dd HH:mm:ss.SSS" ]

      target => "@timestamp"
      remove_field => [ "timestamp", "timestampMs", "logdate" ]
    }
}

output {
    elasticsearch {
      hosts => [ "localhost" ]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
}

First off: the custom pattern "TIMESTAMP_GERMAN" was set in a text file like this:

TIMESTAMP_GERMAN %{MONTHDAY}.%{MONTHNUM}.%{YEAR}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})

And was tested with Grok Matcher for validity. Plus, the Logstash logs (if they're written at all under "/var/log/logstash/") tell me

Duplicate keys found in your configuration: At line: 70, column 16 (byte 1559)

There is not even a "line 70" (my config only has 58 lines), but anyway. I tried fiddling around with this:

grok {
  break_on_match => false
  tag_on_failure => [ ]
  match => [ "message", "%{TIMESTAMP_ISO8601:logdate}[\t ]{1,}%{LOGLEVEL:loglevel}[\t ]{1,}\[%{DATA:thread}\][\t ]{1,}%{DATA:class}\(%{DATA:source_filename}:%{NUMBER:source_linenumber}\):[\t ]{1,}%{GREEDYDATA:message}" ]
  match => [ "message", "%{TIMESTAMP_GERMAN:logdate}[\t ]{1,}%{LOGLEVEL:loglevel}:[\t ]{1,}%{GREEDYDATA}:[\t ]{1,}%{GREEDYDATA:message}" ]
}

But this doesn't work either. What's also very strange is that calling Logstash directly, via

/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/01-beats.conf

does nothing at all. All I see for minutes (I waited 15 minutes for some type of console feedback) is "nothing". Using the same call with Logstash 2.x gives me: "Configuration OK".

It's really troublesome for me right now.
Thank you for your help!