Filebeat multiline and other questions

Elastic Stack Beats
1

Hi,

I am trying to write a jvm verbose gclog filebeat module:

  • where is the right place to set the multiline option?

  • how do I tell elastic to use my pipeline.json with grok filters as ingest pipeline.

  • how to extend the filebeat.template.json and load automatically?

Sorry the how to get started with beats guide is far from complete to answer this all sufficiently.

Thanks for reading, great if you had answers.

Regards,
Eduard

2

Hi!

What version of Filebeat are you using?

  • filebeat/module/yourmodule/yourfileset/config/yourfileset.yml is the file which includes multiline configuration. Where yourmodule is the name of your module and yourfileset is the name of your fileset.
  • If your pipeline is in the standard place, it will load automatically. However, it is not there, you can configure it in the manifest.yml.
  • To generate required files, you need to call make update. After it is generated, Filebeat loads it automatically on startup.

Are you referring to this guide? https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-devguide.html

3

Hi Noémi,

thanks, I think I did that all, but maybe with intermediate steps.
So far it wont eat all lines (only the first) after the negated regex.
Will setup a dev elastic environment and do it like that in one go and see if it works out like that.

Regards,
Eduard

4

There is an extensive guide on multiline and also a Playground where you can test you events: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

If it still does not work, please share your logs and multiline config.

5

Hello Noémi,

I saw that and retestet my multiline setup and looks fine there.

Give me some time to setup an elastic test env and I will let you know if it works doing it all in one go.

Have a nice weekend.

Regards,
Eduard

6

Hi Noémi,

ok the multiline is ok now. But I have another issue, I can open a new thread or just ask: the make update creates a template for the fields, but there is also _meta/fields.yml and it seems those fields get added too!?

Short: how/where to place the fields for my module, and how to add them to the template to be created inside the index(pattern)?

Edit> so adding the new fields to _meta/fields.yml is enough, filebeat adds these to the index/template automagically.

I will do some more tests when done @kvch: where is the best place to propose new modules/comunity beats?
Well mine is more an example, there are some prework to do adopting my verbose gc log parser to another java version or gc type (the output pattern my differ)

Thanks again for your advice.

Regards,
Eduard

7

Yes, indeed.

You can submit your requests here: https://github.com/elastic/beats/issues

8

This topic was automatically closed after 21 days. New replies are no longer allowed.