[processors] required property is missing

Elastic Stack Elasticsearch
1

I am creating a ingest pipeline for custom index for Azure activity log with same configurations it has for Filebeat index. I copied pipeline content from _ingest/pipeline. Due to character limit, I am unable to paste the entire file here.

The error I am facing is :

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "[processors] required property is missing",
        "property_name" : "processors"
      }
    ],
    "type" : "parse_exception",
    "reason" : "[processors] required property is missing",
    "property_name" : "processors"
  },
  "status" : 400
}
2

You can share it on gist.GitHub.com and link here.

3

@dadoonet thanks ! here is the link

4

You can't define 2 pipelines with one call.
You need 2 calls, one for each pipeline.

See Create or update pipeline API | Elasticsearch Guide [7.12] | Elastic

5

I tried creating first one with the following and still getting same error.

PUT _ingest/pipeline/signals-azure-activitylogs-azure-shared-pipeline
{
  "signals-azure-activitylogs-azure-shared-pipeline" : {
    "description" : "Pipeline for parsing azure activity logs.",
    "processors" : [
      {
        "set" : {
          "field" : "cloud.provider",
          "value" : "azure"
        }
      },
      {
        "grok" : {
          "ignore_failure" : true,
          "field" : "azure.resource_id",
          "patterns" : [
            "/SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/NAMESPACES/%{NAMESPACE:azure.resource.namespace}/AUTHORIZATIONRULES/%{RULE:azure.resource.authorization_rule}"
          ],
          "pattern_definitions" : {
            "GROUPID" : ".+",
            "PROVIDERNAME" : ".+",
            "NAMESPACE" : ".+",
            "RULE" : ".+",
            "SUBID" : """(\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}"""
          }
        }
      },
      {
        "grok" : {
          "field" : "azure.resource_id",
          "patterns" : [
            "/SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}"
          ],
          "pattern_definitions" : {
            "SUBID" : """(\{){0,1}[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}(\}){0,1}""",
            "GROUPID" : ".+",
            "PROVIDERNAME" : """([A-Z])\w+.([A-Z])\w+/([A-Z])\w+.""",
            "NAME" : "((?!AUTHORIZATIONRULES).)*$"
          },
          "ignore_failure" : true
        }
      },
      {
        "grok" : {
          "field" : "azure.resource_id",
          "patterns" : [
            "/providers/%{PROVIDER:azure.resource.provider}"
          ],
          "pattern_definitions" : {
            "PROVIDER" : ".+"
          },
          "ignore_failure" : true
        }
      },
      {
        "rename" : {
          "field" : "azure.resource_id",
          "target_field" : "azure.resource.id",
          "ignore_missing" : true
        }
      },
      {
        "lowercase" : {
          "ignore_missing" : true,
          "field" : "event.outcome"
        }
      }
    ],
    "on_failure" : [
      {
        "set" : {
          "field" : "error.message",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }
    ]
  }
}
6

Read carefully the documentation I linked to.

You will see that you must remove:

signals-azure-activitylogs-azure-shared-pipeline" : {
7

Thanks @dadoonet

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.