Ingest pipeline KV processor failure

[tarp (Tim Arp)]

Hi,
I'm running ES 5.4.1, I'm sending a file to my ingest nodes that looks like this.

time:1500651652886|serial:RWGSIPA530083|appName:DataSyncTab|data:This is a string log message
time:1500651652887|serial:RWGSIPA530083|appName:DataSyncTab|page:Opened DataSync Diagnostic Page|status:MTG connection active

My ingest pipeline is this,

PUT _ingest/pipeline/gen4analytics
{
"description": "gen4analytics",
"processors": [
{
"kv": {
"field": "message",
"field_split": "|",
"value_split": ":"
}
},
{
"date": {
"field": "time",
"formats": ["UNIX_MS"]
}
},
{
"date_index_name": {
"field": "@timestamp",
"index_name_format": "yyyy.MM.dd",
"index_name_prefix": "gen4analytics-",
"date_rounding": "d"
}
}
],
"on_failure": [
{
"set": {
"field": "_index",
"value": "failed-{{ _type }}"
}
},
{
"set": {
"field": "error",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}

I'm getting a failure that "field [message] does not contain value_split [:]" .
I don't get it because my field clearly contains ":".
Thanks,
Tim

[talevy (Tal Levy)]

Since these character splits represent regex expressions. the | needs to be escaped
here is an example that illustrates. without the \\, this pipeline runs into the same problem as you mention.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "test colon",
    "processors": [
      {
        "kv": {
          "field": "message",
          "field_split": "\\|",
          "value_split": ":"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": { "message": "time:1500651652887|serial:RWGSIPA530083" }
    }
  ]
}

[tarp (Tim Arp)]

Thank you, that was what I needed.