If condtion not working in ingest pipeline

Elastic Stack Elasticsearch
1

I am trying to set up a ingest pipelines for my Kubernetes cluster logs, but when I try to evaluate a field in the pipeline processor, fields with no dot works ( "if": "ctx.tags == 'audit'") & field with dot characters is not working "if": "ctx?.kubernetes?.container?.name == 'istio-proxy'",. it's strange & would need community advice on the same.

Base pipeline:

PUT _ingest/pipeline/apps-prod-logs-v1
 {
  "processors": [
    {
      "pipeline": {
        "description": "If 'tags' is 'audit', use 'audit pipeline'",
        "if": "ctx.tags == 'audit'",
        "name": "apps-audit-v1"
      }
    },
    {
      "pipeline": {
        "description": "If 'kubernetes.container.name' is 'istio-proxy', use 'istio pipeline'",
        "if": "ctx?.kubernetes?.container?.name == 'istio-proxy'",
        "name": "apps-istio-v1"
      }
    }
  ]
}

input Data sample (not working):


PUT /my-index/_doc/5?pipeline=apps-prod-dps-logs-v1
{
    "kubernetes.container.name": 
      "istio-proxy"
    ,
    "kubernetes.replicaset.name": 
      "test-5986864968"
    ,
    "host.os.codename": 
      "Core"
    ,
    "message": 
      "2021-09-07T11:12:06.457Z \"POST /test/api/v1/default/sessions HTTP/1.1\" 200 - \"-\" \"-\" 531 321 70 70 \"10.220.211.40\" \"Apache-CXF/3.4.0\" \"bf9d5879-1e99-9914-99e7-028ddffe4ef3\" \"test.example.com\" \"127.0.0.1:8081\" inbound|8081|http-app|test.apps-test.svc.cluster.local 127.0.0.1:58524 10.220.79.49:8081 10.220.211.40:0 outbound_.8081_._.test.apps-test.svc.cluster.local default"
    ,
    "kubernetes.node.labels.kubernetes_io/hostname": 
      "ip-10-000-00-00.ap-south-2.compute.internal"
    ,
    "tags": 
      "apps"
    
    
  }

output

GET /my-index/_doc/5

not working output

{
  "_index" : "my-index",
  "_type" : "_doc",
  "_id" : "5",
  "_version" : 1,
  "_seq_no" : 33,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "kubernetes.container.name" : "istio-proxy",
    "kubernetes.replicaset.name" : "test-5986864968",
    "host.os.codename" : "Core",
    "message" : """2021-09-07T11:12:06.457Z "POST /test/api/v1/default/sessions HTTP/1.1" 200 - "-" "-" 531 321 70 70 "10.220.211.40" "Apache-CXF/3.4.0" "bf9d5879-1e99-9914-99e7-028ddffe4ef3" "test.example.com" "127.0.0.1:8081" inbound|8081|http-app|test.apps-test.svc.cluster.local 127.0.0.1:58524 10.220.79.49:8081 10.220.211.40:0 outbound_.8081_._.test.apps-test.svc.cluster.local default""",
    "kubernetes.node.labels.kubernetes_io/hostname" : "ip-10-000-00-00.ap-south-2.compute.internal",
    "tags" : "apps"
  }
}

Working input (tags: audit)

PUT /my-index/_doc/3?pipeline=apps-prod-dps-logs-v1
{
    "tags": 
      "audit"
    ,
    "message": 
      """2021-09-07 08:21:00,529|INFO|main|com.test.test.data.test|date=2021-09-07T08:21:00.519Z,user=test,event=start,success=true,origin=127.0.0.1,resource=test,groups="""
}

GET /my-index/_doc/3

#output


{
  "_index" : "my-index",
  "_type" : "_doc",
  "_id" : "3",
  "_version" : 12,
  "_seq_no" : 34,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "date" : "2021-09-07 08:21:00,529",
    "level" : "INFO",
    "log_msg" : "date=2021-09-07T08:21:00.519Z,success=true,origin=127.0.0.1,resource=test,groups=",
    "thread" : "main",
    "message" : "2021-09-07 08:21:00,529|INFO|main|com.test.test.data.test|date=2021-09-07T08:21:00.519Z,user=system,event=start,success=true,origin=127.0.0.1,resource=test",
    "tags" : "audit",
    "processed_pipleline" : "apps-audit",
    "dps-audit" : {
      "date" : "2021-09-07T08:21:00.519Z",
      "resource" : "test",
      "success" : "true",
      "origin" : "127.0.0.1"
    },
    "class" : "com.test.test.data.test"
  }
}

Thanks
Venkat

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.