Module create Logstash filter?

Elastic Stack Beats
1

We would like to use Modules to instantiate logging for our services. IE - We deploy a service and it deploys an associated Filebeat module that has it's own specific log paths and creates the needed Kibana dashboards. Seems like an awesome way to encapsulate things.

EXCEPT - We have some services that output XML. From what I've read, the easiest way to parse XML is to use the XML filter in Logstash. Modules can insert dashboards into Kibana, is there a way to insert filters into Logstash? If not, is the expectation that the services' log parsing is done via ingest/pipeline.json and pushed directly into Elasticsearch? If so, is there a Filebeats-side XML parser?

What would be the best way to approach this?

2

You can use a filebeat module to insert filters and dashboards (etc), but you only need to do that once, if you give your source nodes write access to your logstash and elastic instances.

Since your nodes would all be using the same dashboard and filters, I'd recommend maintaining that in a separate project and updating it as needed. No sense in installing it every time a client gets installed. (actually, pushing that to elastic/logstash is an optional step in the beats install, normally you'd run it from a controlled machine once and your sources wouldn't see it.)

For your XML, you'll want a second module (or prospector, those are easier), that labeled those files so that your filters would route them appropriately.

3

You can use a filebeat module to insert filters and dashboards (etc)

The heart of my problem: I was unable to find any documentation on how to use Filebeat to insert a Filter (I assume you meant Logstash Filter). Do you have any further info?

4

I recommend that you don't. Bundling the install-once items with the install-on-every-node items seems odd to me. I'd keep them as separate packages. It would keep your filters and dashboards from being overwritten every time you install a node. Also Beats prefers to use the Elastic Pipeline, not Logstash.

But, here's the link I know about:

https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.