Custom Module Issue - Ingest Pipeline Alpha Sorted?

Elastic Stack Beats
1

I have created a custom module using make create-fileset. I've populated the various files, run make collect, and copied the results into my filebeat module directory. ALMOST everything is working as expected.

When filebeat executes the first time, it loads the ingest pipeline in alphabetical order rather than the order set in the pipeline.json file*. This causes the grok processor, that needs to be processed first, to execute after the other processors.

Has anyone seen this before? is there a way to force filebeat to respect the ordering in the pipeline.json file when pushing the pipeline to ES?

Configuration:
elastic search & kibana 5.4.1 (via docker-compose up)
filebeat 5.5.0 (installed via brew)

  • I've confirmed the pipeline processor ordering is correct in the module's pipeline.json and that filebeat, with debug out, is reading the pipeline and pushing it in alphabetical order.
2

Answering my own post... This took quite a while to figure out. (Probably more than it should have :frowning: )

When initially loading a pipeline from a module, filebeat alphabetizes the processor entries within each ingest sub-block. Filebeat doesn't adhere to the ordering within the pipeline.json file.

To ensure proper ordering and execution, place any directives to be processed in sequential order in discrete {} blocks (nested at the same level).

In my case, the initial pipeline.json file was formatted as:

{
  "description" : "my pipeline",
  "processors" :  [
    {
      "grok": {
          ...
      },
      "date" : {
        ...
      }
    }
  ]
}

After isolating each processor in a discrete block, filebeat loaded the pipeline as intended :

{
  "description" : "my pipeline",
  "processors" :  [
    {
      "grok": {
          ...
      }
    },
    {
      "date" : {
        ...
      }
    }
  ]
}