How do I fix "Invalid FieldReference" errors?

Elastic Stack Logstash
1

Logstash is reporting errors like this every second or so:

[2021-09-10T19:09:56,647][INFO ][org.logstash.beats.BeatsHandler][main][98ee7b7f6aa150efeb06cd1c765e51dc0b1e1cd81424a2538524f239b81fa590] [local: 10.0.3.241:5044, remote: 10.0.3.4:41652] Handling exception: org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_seedbox-bridgetroll_tls_domains[0]_sans` (caused by: org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_seedbox-bridgetroll_tls_domains[0]_sans`)
[2021-09-10T19:09:56,647][WARN ][io.netty.channel.DefaultChannelPipeline][main][98ee7b7f6aa150efeb06cd1c765e51dc0b1e1cd81424a2538524f239b81fa590] An exceptionCaught() event was fired, and it reached at the tail of
the pipeline. It usually means the last handler in the pipeline did not handle the exception.
org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_seedbox-bridgetroll_tls_domains[0]_sans`

I am pulling in Docker information with filebeat, including labels for containers in a swarm. One of the containers has labels like this:

      - "traefik.http.routers.seedbox-bridgetroll.tls.domains[0].main=example.com"
      - "traefik.http.routers.seedbox-bridgetroll.tls.domains[0].sans=*.example.com"

What should I do to fix this? I've tried filters like this one, but I'm not sure what to put in instead of [data], and my guesses didn't result in any changes in behavior.

Any help appreciated.

2

I believe that is part of the beats input, so the exception is occurring before the event gets to the pipeline. You will not be able to fix it with a filter.

3

Can I fix it with something else? What is the best way to handle bad beats input?

4

I don't think you can fix it. If the labels are being added by an add_docker_metadata processor then filebeat should be encoding the data in a way that a beat input can interpret. I don't know the details of the lumberjack protocol so I am not sure at which end the problem lies.

5

I see. Does that mean it's a bug in filebeats, then? I am happy to file one if that would be helpful.

6

Well first I would confirm that removing the add_docker_metadata processor causes the error to go away.

Then, is that the label key? If so, it is not valid, Keys can contain lower-case alphanumeric characters, the period character (.), and the hyphen character (-). You could open a bug saying that invalid keys result in exceptions but I doubt it will get much traction.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.