Ingest Node: Fixing Conflicts

[dawiro]

Hi,
Can someone show me how I can convert a simple string to an object using an ingestion pipeline?

For example, I have logs including error objects containing entires such as error.message. And there are other inbound logs containing error strings. How should I go about converting error to error.message so it gets indexed as an object?

Regards,
D

[spinscale (Alexander Reelsen)]

How about this example, where you can check the type of an object and then convert it to a hashmap using a script processor

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "source": "if (ctx.error instanceof Map) { ctx.error.foo = 'bar'; } else if (ctx.error instanceof String) { def message = ctx.error ; ctx.error = [:] ; ctx.error.message = message; }"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "error": "this is a string"
      }
    },
    {
      "_source": {
        "error": {
          "message": "this is an object"
        }
      }
    }
  ]
}

[dawiro]

Thx @spinscale. How will this impact ingestion throughput?

[spinscale (Alexander Reelsen)]

There will be a reduction in throughput as every index operation needs to operate this script. The exact impact in terms of performance or percentage depends highly on your hardware and additional resources available, so it is hard to tell from the outside. The good thing is, that you can scale this part of your indexing separately by using dedicated ingest nodes, that execute the above task independently from indexing.

--Alex

[dawiro]

@spinscale, that looks pretty good thx How do I add a tag to a modified event in one of these scripts so I can track changes?

[dawiro]

Ah, I see there's a tag option to the script processor