Some questions about Filebeat that are not clear in the docs

Elastic Stack Beats
1

Hi team, just some quick questions:

  1. Can Filebeat setup a pipeline in Elasticsearch like it does with the template? What I meant is that I can send a JSON index template from FIlebeat to Elasticsearch using setup.template.json.path: "/etc/filebeat/wazuh-template.json", is there something similar to send ingest pipelines?
  2. How can I tell Filebeat that the time field for reference should be myfield instead of @timestamp? This question is about using %{+yyyy.MM.dd} in the Elasticsearch output, it takes @timestamp but it would be nice if I can change it.
  3. How can I use %{+yyyy.MM.dd} for other fields than @timestamp (related to 2. ) ?
  4. Can I split/extract strings from a field? Use case: my JSOn log has "id": "11:22" how can I extract 11 and 12 into different fields before sending it to Elasticsearch?

Important note 1: I know most things can be achieved using Logstash, this use case has no Logstash.
Important note 2: Some other things can be achieved using some ingest pipes in Elasticsearch, but I want to avoid doing a curl to Elasticseach and leave most of the logic to Filebeat when possible.

Thanks in advance, you guys are doing a great job.

Regards,

2

Any news guys?

3

Hi, jesusgn90! I quickly read your questions, and the answers are yes, it can be done with filebeat

  1. When your write module - filebeat create ingest pipelines
  2. In kibana you can setup timestamp field, in elastic - its template (set correct type to field)
  3. templates (mappings)
  4. yeah, its module (regexp, patterns)
4

although, after reading more carefully, need to allocate time and answer them ... my answer is higher, in some points it is not quite correct.

5

  1. What @maxozerov said about this is correct. This is only possible with modules today. For instance, the apache module defines an ingest pipeline for parsing Apache HTTP server access logs here: https://github.com/elastic/beats/tree/master/filebeat/module/apache/access/ingest.

    To offer this as a "top-level" or "global" feature in Filebeat might be useful. To be honest, I'm not entirely sure why this is not the case already. If you have a GitHub account, I'd suggest opening an enhancement request for this feature here: https://github.com/elastic/beats/issues/new?template=feature-request.md. That way you can elaborate on your use case and follow along with the discussion and progress on the issue. If you don't have a GitHub account, let me know and I'll make an issue for you.

  2. @timestamp is a fundamental and, in fact, required field in all Beats events. There is no way to override it. Given that, perhaps you could just copy the value of myfield to the @timestamp field? Perhaps https://www.elastic.co/guide/en/elasticsearch/reference/master/date-processor.html could help?

  3. See 2.

  4. Perhaps https://www.elastic.co/guide/en/elasticsearch/reference/master/split-processor.html could help?

Cheers,

Shaunak

6

Hi @maxozerov, @shaunak,

Thanks for the replies.

  1. I thought the same, then we need an improvement for this, it would be great, there are multiple use cases.
    2, 3 and 4. Yes, processors and more processors, so the question here is: Is there a way to have a processor in Elasticsearch without doing a curl to enable it?

Regards

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.