Field [ts] of type [java.lang.Double] cannot be cast to [java.lang.String]

Elastic Stack Elasticsearch
,
1

Hi All,

I have few application in kubernetes environment which log timestamp field as "ts" and in unix epoch timestamp (in seconds.milliseconds format). i am referring this post to use ingest pipeline grok processor to extract value and convert it to proper date/time format. Below is my attempt using ingest pipeline simulate feature and it failes with error: field [ts] of type [java.lang.Double] cannot be cast to [java.lang.String]. i will really appreciate any help to fix this.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field" : "ts",
          "patterns": ["\\[%{NUMBER:epoch_ts}]"],
          "pattern_definitions" : {
            "TIMESTAMP" : "[0-9]+"}
        }
      },
      {
      "date":{
        "field":"epoch_ts",
        "formats":["UNIX"]
      }
      }
    ]
  },
  "docs": [
    {
      "_source":{
      "level": "info",
  "ts": 1683017794.2955067,
  "logger": "controller",
  "msg": "handling constraint update",
  "process": "constraint_controller",
  "instance": {
    "apiVersion": "constraints.gatekeeper.sh/v1beta1",
    "kind": "K8sAzureV1BlockDefault",
    "name": "azurepolicy-k8sazurev1blockdefault-13e93bddde21c320458a"
  },
  "stream": "stderr",
  "logtag": "F",
  "docker": {
    "container_id": "bba7724c469da3e6efc2f4690cf88dd15378cb138fa51bac83395c4930af5b26"
  },
  "kubernetes": {
    "container_name": "gatekeeper-controller-container",
    "namespace_name": "gatekeeper-system",
    "pod_name": "gatekeeper-controller-74559c67f6-zgw2c",
    "container_image": "mcr.microsoft.com/oss/open-policy-agent/gatekeeper:v3.11.0",
    "container_image_id": "sha256:f8e149e44d6efa24f18f4dcd391fe462a6023c1ffc97bda2e61c456a7d7eaeef",
    "pod_id": "a986e78b-bb8e-4136-ad54-ca9fa6e875e5",
    "pod_ip": "172.19.54.64",
    "host": "aks-nodepool1-30653362-vmss000001",
    "labels": {
      "control-plane": "controller-manager",
      "gatekeeper_sh/operation": "webhook",
      "gatekeeper_sh/system": "yes",
      "pod-template-hash": "74559c67f6"
    },
    "master_url": "https://10.0.0.1:443/api",
    "namespace_id": "65825090-5942-4301-a780-12205d64dd60",
    "namespace_labels": {
      "addonmanager_kubernetes_io/mode": "Reconcile",
      "admission_gatekeeper_sh/ignore": "no-self-managing",
      "control-plane": "controller-manager",
      "gatekeeper_sh/system": "yes",
      "kubernetes_io/metadata_name": "gatekeeper-system"
    }
  },
  "@timestamp": "2023-05-02T08:56:34.882460076+00:00",
  "tag": "kubernetes.var.log.containers.gatekeeper-controller-74559c67f6-zgw2c_gatekeeper-system_gatekeeper-controller-container-bba7724c469da3e6efc2f4690cf88dd15378cb138fa51bac83395c4930af5b26.log"
    }
    }
    ]
}
2

i tried using below single processor which probably fixing above error but it is causing to throw new error .

"pipeline": {
    "processors": [
      {
      "date":{
        "field":"ts",
        "formats":["UNIX"]
      }
      }
    ]
  },

new error on similar data:
[error type]: illegal_argument_exception [reason]: 'unable to parse date [2023-05-02T10:48:49.132335179Z]'

3

Per Date processor | Elasticsearch Guide [8.7] | Elastic you can specify multiple formats;

An array of the expected date formats

So maybe try "formats":["UNIX", "ISO8601"]?

4

Hi @warkolm Thanks for your response, i tried other formats but they did not help. somehow UNIX format worked next day, probably due to index rollover and mappings i modified on index template.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.