Ingest-converter not mapping some fields properly?

[Daniel314 (Daniel C)]

Background: at my organization, we heavily use Logstash and kafka to deal with buffering and spikes in the traffic flowing to ElasticSearch, as well as scaling processing horizontally.

Because of this, the "ingest" pipelines in Filebeats aren't directly useful. So I've investigated the ingest-convert.sh script that comes with Logstash, and mostly have it working (side note: it would be nice if the docs were updated to note that the Beats ingest "logic" is now in YAML instead of JSON like the converter wants).

However, I've found that some fields don't appear to be getting correctly parsed by the converter, and that's the reason for this post.

When I convert the Filebeat iis access ingest module (Filebeat v7.10.2) using the Logstash 7.10.2 ingest converter, fields enclosed in {{ and }} don't get mapped -- they're passed through verbatim (like "{{source.ip}}"). I'm pretty sure some cleanup of the output is expected, but this particular one can easily become tedious...

Is this expected behavior, to pass "{{field.name}}" through verbatim into the Logstash config? It sure seems like a bug to me.

Thanks,

 - Daniel

[Daniel314 (Daniel C)]

The existing converter seems to be fairly minimal in what it will convert, and it's silent about what it doesn't convert.

I've written my own version of the converter that handles the "if" conditionals, and includes (commented) in the source code the original syntax for unhandled modules. The new version probably isn't perfect (its method of detecting missing fields is flawed), but it still captures considerably more of the original semantic content than the official Logstash ingest converter, and it handles YAML as well as JSON source.

If you find bugs, please let me know. I may or may not additional functionality.

The URL is here: perl script

• Daniel