About Beats Input

Elastic Stack Logstash
1

Hi,
We occasionally receive log lines like this at the beats input:

{"level":"info","ts":1681833658.6787357,"caller":"filterprocessor@v0.75.0/traces.go:74","msg":"Span filter configured","env":"","hostname":"tst","tags":["bad_data"],"kind":"processor","name":"xxxxxx","pipeline":"traces","[Include] match_type":"[None]","[Exclude] match_type":"strict"}

And the input fails with an invalid field reference for "[Include] match_type". Is there documentation to explains what is acceptable or not at the input as this log line is valid json.

Thx
D

2

I cannot answer that question, but this github issue discusses the issue and links to at least one PR that tries to address it.

1 Like
3

That's interesting though its focus is on the json filter plugin. It seems to me that this is actually an upstream issue with the beats (filebeat) client. It should not be parsing and passing json that will break the downstream beats input. When this happens it cripples throughput.

To be clear, this issue is happening at the input. It never reaches the filter stage.

4

Not really, Colin clearly understands that it affects any input/filter that parses valid JSON to create a structure that cannot be encoded into a logstash event.

The PR that it links to makes changes way down in ConvertMap.java, so I would expect that to fix things for anything that creates fields on events.

Try setting config.field_reference.escape_style and see.

1 Like
5

Yes. I see this has a fuller scope that I assumed. Seems like what I need.

6

OK. Well I tried it:

config.field_reference.escape_style: percent

And it didn't work. Logstash version is 8.4.3.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.