Processor -> split -> set array.element?

Elastic Stack Beats
1

I posted a similar question to the elasticsearch forum since it appeared to be more of an intrinsic ES problem. But I am not sure.

I have the following filebeat yaml.

Basically I am attempting to split based on \t (tab) then assign the array output via set.
Just testing the first field right now. But it ends up in ES as an empty "" value.
Is this possible? How do I reference the elements of the array after 'message' is passed to the split processor?

Thanks

{
"description": "OpenAM Authentication Access Logging",
"processors": [{
"set" : {
"field": "type",
"value": "amAuthentication.access_pipeline"
},
"split": {
"field": "message",
"separator": "\t"
},
"set": {
"field": "openam.data",
"value": "{{message[1]}}"
}
}
],
"on_failure": [
{
"set": {
"field": "error",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}

2

For testing/developing an ingest pipeline in Elasticsearch I find the simulate API super helpful: https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html

From Kibana developer console you can test your pipeline via:

POST _ingest/pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": ...
      }
    },
    ... // more samples
  ],
  "pipeline": {
    "description": "",
    "processors": [
      ...
    ]
  }
}

By changing the URL to _ingest/pipeline/_simulate?verbose you will get the result for each intermediary processor.

3

Thanks, I missed the verbose option. Going to try that this morning.

Thank you.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.