Problem with config?

Elastic Stack Logstash
1

Hi,

I've been testing out some grok patterns and I thought I had everything working but in production I get the following output:

+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug.   |
| You can find help with this problem in a few places:    |
|                                                         |
| * chat: #logstash IRC channel on freenode irc.          |
|     IRC via the web: http://goo.gl/TI4Ro                |
| * email: logstash-users@googlegroups.com                |
| * bug system: https://logstash.jira.com/                |
|                                                         |
+---------------------------------------------------------+
The error reported is: 
  undefined group option: /(?<a0>(?<a1>(?<a2>(?:(?<a3>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a4>(?:(?<a5>((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?<a6>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9]))))))(?::(?<a7>\b(?:[1-9][0-9]*)\b))?) [-] (?<a8>(?<a9>[a-zA-Z0-9._-]+)) (?<a10>\[(?<a11>.*?)\]+) "(?:(?<a12>\b\w+\b) (?<a13>\S+)(?: HTTP\/(?<a14>(?:(?<a15>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))))?|(?<a16>.*?))" (?<a17>(?:(?<a18>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))) (?$)/

The config in question is:

input {
  file {
    type => "flowforce"
    path => ["/var/opt/Altova/FlowForceServer2015/data/fflog.log", "/var/opt/Altova/FlowForceServer2015/data/flowforce.log"]
  }
}
filter {
    if [type] == "flowforce" {
      if "_grokparsefailure" in [tags] {
        drop { }
      }
      grok {
        patterns_dir => "/etc/logstash/patterns/flowforce-patterns"
        match => [ "message", "%{FFP1}", "message", "%{FFP2}", "message", "%{FFP3}" ]
      }
      date {
       match => [ "timestamp", "[dd/MMM/yyyy:HH:mm:ss]", "yyyy-MM-dd HH:mm:ss.SSSSSS" ]
      }
    }
}
output {
  redis {
    host => "*****"
    data_type => "list"
    key => "logstash"
    codec => json
  }
}

The patterns referenced are:

FFP1 %{URIHOST:clientip} [-] %{USER:ident} %{SYSLOG5424SD} "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?$
FFP2 %{SYSLOG5424SD:timestamp} %{GREEDYDATA:message}
FFP3 %{TIMESTAMP_ISO8601:timestamp}[|]%{LOGLEVEL:loglevel}[|]%{GREEDYDATA:message}

I ran the config using -t and it said the configuration was OK..

Does anybody have any ideas what might be up?

Thanks

Luke

2

You have (?$ in your FFP1 pattern. Is that really correct?

3

Ah ha. No, it's very much incorrect. It's truncated.

Well spotted. Thank you very much.

Luke