How come my ElasticSearch ingest pipelines with multiple grok patterns are all failing?

[ujjain]

I'm working on getting the Tomcat logs of some applications correctly processed in ElasticSearch, but unfortunately my ingest pipelines with multiple grok processors aren't working and they all end up in the failed-index.

catalina.out

13-Feb-2018 16:04:48.446 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxGCPauseMillis=200

catalina.log

13-Feb-2018 16:00:35.385 SEVERE [monkey.co.uk-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Error deploying web application archive [/opt/tomcat/apache-tomcat-8.5.16/webapps/email.war]
 java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[monkey.co.uk].StandardContext[/email]]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:756)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

This is my ingest pipeline:

PUT _ingest/pipeline/tomcat_all

{  
   "description":"Pipeline tomcat_all",
   "on_failure":[  
      {  
         "set":{  
            "field":"_index",
            "value":"failed-{{ _index }}"
         }
      }
   ],
   "processors":[  
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{MONTH}%{SPACE}%{MONTHDAY},%{SPACE}%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}(?:AM|PM)%{SPACE}%{NOTSPACE:class}%{SPACE}%{NOTSPACE:type_log}%{SPACE}%{WORD:loglevel}:%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{TIME:timestamp}%{SPACE}\\|-%{WORD:loglevel}%{SPACE}in%{SPACE}%{NOTSPACE:class}%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{MONTHDAY}[/-]%{MONTH}[/-]%{YEAR} %{TIME}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{MONTH}%{SPACE}%{MONTHDAY},%{SPACE}%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}(?:AM|PM)%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}%{ISO8601_TIMEZONE}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      },
      {  
         "grok":{  
            "field":"message",
            "patterns":[  
               "%{MONTHDAY}.%{MONTHNUM}.%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}"
            ]
         }
      }
   ]
}

Grok patterns:

%{MONTH}%{SPACE}%{MONTHDAY},%{SPACE}%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}(?:AM|PM)%{SPACE}%{NOTSPACE:class}%{SPACE}%{NOTSPACE:type_log}%{SPACE}%{WORD:loglevel}:%{SPACE}%{GREEDYDATA:log_text}
%{TIME:timestamp}%{SPACE}\\|-%{WORD:loglevel}%{SPACE}in%{SPACE}%{NOTSPACE:class}%{SPACE}%{GREEDYDATA:log_text}
"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}
%{MONTHDAY}[/-]%{MONTH}[/-]%{YEAR} %{TIME}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}
%{MONTH}%{SPACE}%{MONTHDAY},%{SPACE}%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}(?:AM|PM)%{SPACE}%{GREEDYDATA:log_text}
%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}
%{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}%{ISO8601_TIMEZONE}%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}
%{MONTHDAY}.%{MONTHNUM}.%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}%{WORD:loglevel}%{SPACE}%{GREEDYDATA:log_text}

I have tried the grok patterns manually at http://grokdebug.herokuapp.com and that worked fine.