Multiple Grok Processors in pipeline: "[processors] required property is missing"

[KookieMeister (Kris)]

Hi there,

I'm currently using the following pipeline with success:

{
    "ingestpipeline" : {
      "description" : "...",
      "processors" : [
        {
          "grok" : {
            "field" : "message",
            "patterns" : [
              "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
            ],
            "on_failure" : [
              {
                "grok" : {
                  "field" : "message",
                  "patterns" : [
                    "(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
                  ],
                  "on_failure" : [
                    {
                      "grok" : {
                        "field" : "message",
                        "patterns" : [
                          "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
                        ],
                        "ignore_failure" : true
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  }

After using that, I decided to add a second grok processor in the pipeline for extraction from a different source field: log. But whenever I try to use that new pipeline config, I'm given the error message:

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "[processors] required property is missing",
        "property_name": "processors"
      }
    ],
    "type": "parse_exception",
    "reason": "[processors] required property is missing",
    "property_name": "processors"
  },
  "status": 400
}

Here is the pipeline configuration I'm trying to use:

{
    "ingestpipeline" : {
      "description" : "...",
      "processors" : [
        {
          "grok" : {
            "field" : "message",
            "patterns" : [
              "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
            ],
            "on_failure" : [
              {
                "grok" : {
                  "field" : "message",
                  "patterns" : [
                    "(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
                  ],
                  "on_failure" : [
                    {
                      "grok" : {
                        "field" : "message",
                        "patterns" : [
                          "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
                        ],
                        "ignore_failure" : true
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        {
            "grok" : {
                "field" : "log",
                "patterns" : [
                  "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rror|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
                ],
                "on_failure" : [
                  {
                    "grok" : {
                      "field" : "log",
                      "patterns" : [
                        "(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
                      ],
                      "on_failure" : [
                        {
                          "grok" : {
                            "field" : "log",
                            "patterns" : [
                              "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rror|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
                            ],
                            "ignore_failure" : true
                          }
                        }
                      ]
                    }
                  }
                ]
            }
        }
      ]
    }
  }

I'm using AWS Elasticsearch Service OpenDistro version 7.1

[KookieMeister (Kris)]

I'm going to close this issue. It turned out that I was adding in an additional field in the file that I didn't need, the ingestpipeline name with definition. I was using GET to get the current pipeline but was not removing the name definition before PUTing the new modified version. It should have looked like the below:

{
      "description" : "Ingest pipeline for all incoming elasticsearch data",
      "processors" : [
        {
          "grok" : {
            "field" : "message",
            "patterns" : [
              "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
            ],
            "on_failure" : [
              {
                "grok" : {
                  "field" : "message",
                  "patterns" : [
                    "(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
                  ],
                  "on_failure" : [
                    {
                      "grok" : {
                        "field" : "message",
                        "patterns" : [
                          "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
                        ],
                        "ignore_failure" : true
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        {
          "grok" : {
            "field" : "log",
            "patterns" : [
              "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ii]nfo|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rror|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
            ],
            "on_failure" : [
              {
                "grok" : {
                  "field" : "log",
                  "patterns" : [
                    "(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
                  ],
                  "on_failure" : [
                    {
                      "grok" : {
                        "field" : "log",
                        "patterns" : [
                          "(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ii]nfo|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rror|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
                        ],
                        "ignore_failure" : true
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      ]
    }