Field reference from _source for conditional output

Elastic Stack Logstash
1

hello, I'm new to logstash conditional, I want to make different index output based on some field reference on my logstash

here is my index example :

{
  "_index": "iris-new-2021.09",
  "_type": "_doc",
  "_id": "EKS5EHwBUrXRxI7i7tvA",
  "_version": 1,
  "_score": null,
  "_source": {
    "input": {
      "type": "syslog"
    },
    "@timestamp": "2021-09-23T03:36:30.421Z",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "openshift": {
      "message": "\tat com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:554)",
      "level": "unknown",
      "viaq_msg_id": "YTlkYzdhNzItM2U2NC00NmRiLWFmZTItNDc2M2FiZWExZTJj",
      "kubernetes": {
        "container_image_id": "image-registry.openshift-image-registry.svc:5000/iris-uat/iris-batch-swcon-prod@sha256:d4d74861c96f728af3cb953aa77e0f4c7d220541fb82a048a76f546b4e5a3c94",
        "pod_id": "96a3657c-a2e7-4918-9f83-10be97123270",
        "container_name": "iris-batch-swcon-prod",
        "master_url": "https://kubernetes.default.svc",
        "container_image": "image-registry.openshift-image-registry.svc:5000/iris-uat/iris-batch-swcon-prod@sha256:d4d74861c96f728af3cb953aa77e0f4c7d220541fb82a048a76f546b4e5a3c94",
        "host": "devocpworker04.ocpdev.dti.co.id",
        "pod_name": "iris-batch-swcon-prod-1-s57zc",
        "flat_labels": [
          "app=iris-batch-swcon-prod",
          "deployment=iris-batch-swcon-prod-1",
          "deploymentconfig=iris-batch-swcon-prod"
        ],
        "namespace_name": "iris-uat",
        "namespace_id": "0da8b310-ceaf-4446-a645-11f3b792d572"
      },
      "pipeline_metadata": {
        "collector": {
          "ipaddr4": "10.58.81.34",
          "name": "fluentd",
          "version": "1.7.4 1.6.0",
          "received_at": "2021-08-01T03:23:39.492821+00:00",
          "inputname": "fluent-plugin-systemd"
        }
      },
      "docker": {
        "container_id": "5001c9085439a436eb3ef1f146b0e390003e01f93dda1a269b9d2fae3a8a4286"
      },
      "hostname": "devocpworker04.ocpdev.dti.co.id",
      "@timestamp": "2021-08-01T03:23:38.794143+00:00"
    }
  },
  "fields": {
    "openshift.@timestamp": [
      "2021-08-01T03:23:38.794Z"
    ],
    "openshift.pipeline_metadata.collector.received_at": [
      "2021-08-01T03:23:39.492Z"
    ],
    "@timestamp": [
      "2021-09-23T03:36:30.421Z"
    ]
  },
  "sort": [
    1632368190421
  ]
}

I want to use [_source][openshift][kubernetes][namespace_name] as reference to make index output

here's what I've tried :
1.

if "iris-uat" in "%{[openshift][kubernetes][namespace_name]}" {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-uat-%{+YYYY.MM}"
		}
	}

if "iris-uat" in "[openshift][kubernetes][namespace_name]" {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-uat-%{+YYYY.MM}"
		}
	}

if "iris-uat" in [openshift][kubernetes][namespace_name] {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-uat-%{+YYYY.MM}"
		}
	}

am I missing something, I've also tried delete the [openshift] and only use the rest of sub field instead, but I still couldn't get the output

feel free to ask for more information regarding my question if you don't understand about what I'm asking

Thanks

2

if "iris-uat" in "%{[openshift][kubernetes][namespace_name]}" {

You cannot use a sprintf reference in a conditional. This is a substring match.

if "iris-uat" in "[openshift][kubernetes][namespace_name]" {

That is a also substring match against a string, so if "ift][kuber" in "[openshift][kubernetes][namespace_name]" { would evaluate to true, but that never will.

if "iris-uat" in [openshift][kubernetes][namespace_name] {

I would expect that to work. Instead of showing us the result from an index search can you show us an event output from logstash with

output { stdout { codec => rubydebug } }
1 Like
3

fyi, I make a else conditional to output to other index, from every conditional I've used, it is forwarded to the "else" condition, so to make it clear, the "if" condition was not met

4

Yes, I understood that. It is an unexpected result. That is why I asked for additional data.

5

I used the bottom one from you've recommended,
sorry if my information before is not complete enough
here was my config :

if "iris-uat" in [openshift][kubernetes][namespace_name] {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-uat-%{+YYYY.MM}"
		}
	}
else if "iris" in [openshift][kubernetes][namespace_name] {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-dev-%{+YYYY.MM}"
		}
	}

I think the anomaly happened because "iris" and "iris-uat" is counted as same because I used "in" conditional,
rather I use "==" conditional to add tag and then use "in" conditional for the output

here's my code on my filter section:

		if [openshift][kubernetes][namespace_name] == "iris" {
			mutate { add_tag => "iris" }
		} else if [openshift][kubernetes][namespace_name] == "iris-uat" {
			mutate { add_tag => "iris-uat" }
		}

and this on my output section :

if "iris" in [tags] {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-dev-%{+YYYY.MM}"
		}
	}
	else if "iris-uat" in [tags] {
		elasticsearch {
			hosts => ["xxx:9200"]
            index => "iris-uat-%{+YYYY.MM}"
		}
	}
    else {
        #stdout { codec => rubydebug }
        elasticsearch {
            hosts => ["xxx:9200"]
            index => "iris-new-%{+YYYY.MM}"
        }
    }

Now it is fixed, thanks for your explanation, because now I can clearly know how the syntax work

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.