Ingest pipeline context modification inconsistent

Elastic Stack Elasticsearch
1

I'm trying to setup an ingest pipeline that enhances the metricbeat dataset.

Here are two example processors that expose the peculiarity (this is a test setup):

  "script": {
    "inline": "if (ctx.system.process.username ==~ /tomcat/) { ctx.process_name = 'tomcat'}"
  },
  "set": {
    "field": "system.process.name",
    "value": "{{process_name}}"
  },
  "rename": {
    "field": "process_name",
    "target_field": "system.process.name"
  }

I would expect system.process.username to become m but it actually becomes the empty string.

I see two issues here:

  1. the template expression seems to consider only the incoming field values. No modifications are relfected in the set value.
  2. The rename does not 'see' the set value either. The pipeline fails with field [process_name] doesn't exist.

Am I doing it wrong or there is a bug here?

2

Can it be that the script processor can appear only once in the pipeline? It seems like only the last inline script is actually ran. With these processors

  "script": {
    "inline": "ctx.tutu1 = 'sdfsdf'"
  },
  "script": {
    "inline": "ctx.tutu3 = 'sdfsdf'; ctx.tutu2 = 'sdfsdf'"
  }

the doc will contain tutu3 and tutu2, and even if I put an error in the first script, the pipeline will succeed.

Can it be that only the last script is cached and compiled?

3

Ok, I made the mistake. https://github.com/elastic/elasticsearch/issues/25906#issuecomment-318050169

The script started working as soon as I modified it to be a series of processors:

{
  "script": {
    "inline": "ctx.tutu1 = 'sdfsdf'"
  }
},
{
  "script": {
    "inline": "ctx.tutu3 = 'sdfsdf'; ctx.tutu2 = 'sdfsdf'"
  }
}

The JSON format does not make it easy to see the difference.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.