Selectively dedot some fields

[Todd_Lyons]

TL;DR: How to detect subfields with dots, converting those dots to underscores, but ONLY in fields under that subfield.

Detail:
We use filebeats, logstash, and feed messages to an Elastic Stack, of which all pieces are running logstash 6.4.3. We have some Kubernetes deployments that are a mixture of home-grown apps and some general community apps.

When we first started with Kube a couple years ago, we standardized our kube deployment templates with a label "app" to be a text string, with differing values depending on the application it was on. However, modern Kube app deployments are standardizing on labels like:

metadata.labels."app.kubernetes.io/foo"="bar"

The dynamic discovery of this pod and its labels by filebeats results in the following construction in the document:

kubernetes.pod.labels.app.kubernetes.io/foo=bar

This is a problem for us when the final logstash tries to insert the JS document into ES because ES sees the "kubernetes.pod.labels.app.kubernetes.io/foo" and tries to create an object under "app" but we already have defined that "app" must be a string. To be clear, the data coming out of logstash is, I think:

[kubernetes][pod][labels][app.kubernetes.io/foo] = "bar"

but ES converts the dots to field name separators so tries to create an object and index it at:

[kubernetes][pod][labels][app][kubernetes][io/foo] = "bar"

In a different part of the company, a different team has an ELK stack and uses fluentd to get container logs. Something in their config is automatically "dedot"ing the label portion of the field, so what gets inserted into their ES cluster is:

kubernetes.pod.labels.app_kubernetes_io/foo=bar

This would be an ideal solution for us, but I have been struggling to find a way to detect that field (and all fields kubernetes.pod.labels.*) with the dots in it and convert the dots to underscores before it tries to insert the JS document to ES.

I can't run dedot() on it, even passing in the [kubernetes][pod][labels] prefix because dedot doesn't let me specify what level to start dedot-ing at. It will convert the whole object to a single large field, which is not desired. It will look like this :

kubernetes_pod_labels_app_kubernetes_io/foo

I strongly suspect that I will need to use some kind of inline ruby to do this, kind of like what is shown in: https://stackoverflow.com/a/37617214/611911

In my head I'm thinking to loop over [kubernetes][pod][labels], and then in the inline ruby check for each label:

  # if (label.include?("."))       # Would this work as expected?
  if (label =~ /\./)               # Example from the StackOverflow post
    newlabel = label.gsub(".", "_")
    event.set(newlabel, event.remove(label))
  end

Is there anybody who has done this before, either in filebeat or in logstash or in logstash with inline ruby? I'd appreciate a hint (or more!) or example.

[Todd_Lyons]

One thing might work against me is if my understanding is incorrect and the fields are already split on the dot before it gets to my logstash pipeline. It's not clear to me where the app.(Object) is created in the pipeline (where in the lifetime of the JS doc that's being processed).

[Todd_Lyons]

We had a log event today where we lost a bunch of logs due to this conflict. After googling a bit more, I stumbled across a filebeat issue. This issue is about the same thing I was struggling with, and details that filebeat 6.7.0 had a PR which added some options to dedot them by the processor which pulls in kube labels:

[filebeat] Failed to parse kubernetes.labels.app

So my biggest issue was that I am on an older version of filebeats (6.4.3). I am about to test now with filebeat-6.7.2 with those options to see if I can use that to discover container logs and send them to our 6.4.3 logstash relays and 6.4.3 ES nodes. (I suspect that will be fine, but gotta test to be sure).

[Todd_Lyons]

Update: filebeat-6.7.2 did properly mutate the labels and I thought I was going to be successful. But it also does something weird with the message field. I turned on a debug mode for one of our relays and could see the labels getting properly de-dotted. The number of these messages went to zero:

"failed to parse [kubernetes.labels.app]"
...
"Can't get text on a START_OBJECT"

But a very unexepected and confusing behavior started: The "message" field seems to be an object now with filebeat-6.7.2 because my logstash pods which insert into ES began spitting out this error:

"reason"=>"failed to parse [message]",
"caused_by"=>{
  "type"=>"illegal_state_exception",
  "reason"=>"Can't get text on a START_OBJECT at 1:820"
}

So I had to roll them all back to 6.4.3 for now until I can figure out what has changed behaviorally. We're living with the lost messages with those labels until I can find some time to troubleshoot through this.