Rename field depending on its data type

Elastic Stack Elasticsearch
,
1

Hi,

We are currently having issues with the ingestion of some documents due to the data type for one field (log.context.exception). Most of the logs come with this field as an object and the mapping is prepared to store them in the index. But for certain cases, the field comes as string type and the ingestion fails.

I attach both examples of logs and then I explain the methods I tried to solved that in the ingestion pipeline.

Failed log:

{
...
  "log": {
  	"context": {
    	"application_name": "test",
    	"host": "test-site-8752-hjy6",
    	"env": "prod",
    	"remote_ip": "127.0.0.1",
    	"exception": "TypeError: Return value of App\\Ad\\Domain\\Ad\\LinkAd::getPopunder() must be of the type bool, "
  	}
  }
}

Ingested log:

{
...
  "context": {
    "application_name": "test",
    "host": "test-site-8752-hjy6",
    "env": "prod",
    "remote_ip": "127.0.0.1",
    "exception": {
      "class": "Cmp\\Token\\Domain\\Exception\\ExpiredTokenException",
      "message": "",
      "code": "0",
      "file": "/home/wwwroot/test/vendor/cmp/token/src/Cmp/Token/Infrastructure/Test.php:90"
    }
  }
}

Methods:

  1. If log.context.exception doesn't start with { , then rename it to log.context.exception.text :
{
  "rename": {
    "field": "log.context.exception",
    "target_field": "log.context.exception.text",
    "if" : "! ctx.log?.context.exception.startsWith('{')"
  }
}

I saw that ctx.log?.context.exception wasn't working. When I use two subfields it stops working or at least I don't know how to make it work. Cause if I would use just one subfield (ctx.log?.context) then it works. Maybe someone could help me with that.

  1. I tried using painless language. If log.context.exception is not an object, then move it to ctx.log?.context.exception.text. I'm not familiar with painless language so I wasn't 100% sure how to use it for this case. Below you can see what I did.
{
  "script": {
    "lang": "painless",
    "inline": "if (!(ctx.log.context.exception instanceof Map)) { ctx.log.context.exception.text = ctx.log.context.exception } "
  }
}

If some knows how to help me I will be very happy :slight_smile:

Thanks in advance

Mario

2

Hi @mmartinez .

This exemple set for another field if the exception is String.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "description": "Extract 'tags' from 'env' field",
          "lang": "painless",
          "source": """
            if(ctx['log'].context.exception instanceof String) 
            {
              ctx['log.context.exception.text'] = ctx['log'].context.exception;
            } else {
              ctx['log.context.exception'] = ctx['log'].context.exception;
            }
          """,
          "params": {
            "delimiter": "-",
            "position": 1
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "log": {
          "context": {
            "application_name": "test",
            "host": "test-site-8752-hjy6",
            "env": "prod",
            "remote_ip": "127.0.0.1",
            "exception": "TypeError: Return value of App\\Ad\\Domain\\Ad\\LinkAd::getPopunder() must be of the type bool, "

          }
        }
      }
    }
  ]
}
3

Hi @RabBit_BR

You made my day! Thanks for your help.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.