Removing backslashes (\)

[CodeMonky (Name)]

Good day all!

I have log files that I'm trying to ingest into a test pipeline. I have my grok patterns and everything but when I run the pipeline it fails. I believe it is failing due to the logs having back-slashes in the log.

Ex: subtype="ips" eventtype="signature" level="alert" vd="root" etc

I'm trying to use the Script Processor with the following script:

PUT _ingest/pipeline/Test2
{
  "description": "Updated pipeline with changes",
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": "ctx.temp = ctx.temp.replaceAll('\\\\\\\\', '').replaceAll('\\\\', '')"
      }
    }
  ]
}

When I run the processor, it keeps getting a compile error and I'm not sure where. My one clue is this:

But I can't get a real answer on WHY this is wrong. Any Ideas?

Note that I'm using GREEDYDATA to hold everything past the first few pattern parts in a location called temp, which is why you'll see that "temp" used in the script.

Grok: %{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{WORD:device} %{GREEDYDATA:temp}

Thanks!

[jsanz (Jorge Sanz)]

Hi @CodeMonky, welcome to our community. You posted something that is more suited to the Elasticsearch forum (nothing specific about Kibana in your question).

Have you experimented just with the pipeline with the simulate endpoint to isolate the script?

I ran this super simple test in the Dev Console using the gsub processor instead, and it is working well for the tests documents I passed:

POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "gsub": {
          "field" : "temp",
          "pattern" : "\\\\",
          "replacement": ""
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "temp": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "temp": "\\this has \\back \\slashes"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "temp": "-\\-\\\\-\\-"
      }
    }
  ]
}

Resulting with:

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_id": "id",
        "_version": "-3",
        "_source": {
          "temp": "bar"
        },
        "_ingest": {
          "timestamp": "2023-07-20T14:12:56.75455216Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_id": "id",
        "_version": "-3",
        "_source": {
          "temp": "this has back slashes"
        },
        "_ingest": {
          "timestamp": "2023-07-20T14:12:56.754585461Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_id": "id",
        "_version": "-3",
        "_source": {
          "temp": "----"
        },
        "_ingest": {
          "timestamp": "2023-07-20T14:12:56.754639162Z"
        }
      }
    }
  ]
}