Some questions about Filebeat that are not clear in the docs

Hi team, just some quick questions:

  1. Can Filebeat setup a pipeline in Elasticsearch like it does with the template? What I meant is that I can send a JSON index template from FIlebeat to Elasticsearch using setup.template.json.path: "/etc/filebeat/wazuh-template.json", is there something similar to send ingest pipelines?
  2. How can I tell Filebeat that the time field for reference should be myfield instead of @timestamp? This question is about using %{+yyyy.MM.dd} in the Elasticsearch output, it takes @timestamp but it would be nice if I can change it.
  3. How can I use %{+yyyy.MM.dd} for other fields than @timestamp (related to 2. ) ?
  4. Can I split/extract strings from a field? Use case: my JSOn log has "id": "11:22" how can I extract 11 and 12 into different fields before sending it to Elasticsearch?

Important note 1: I know most things can be achieved using Logstash, this use case has no Logstash.
Important note 2: Some other things can be achieved using some ingest pipes in Elasticsearch, but I want to avoid doing a curl to Elasticseach and leave most of the logic to Filebeat when possible.

Thanks in advance, you guys are doing a great job.

Regards,

Any news guys?

Hi, jesusgn90! I quickly read your questions, and the answers are yes, it can be done with filebeat

  1. When your write module - filebeat create ingest pipelines
  2. In kibana you can setup timestamp field, in elastic - its template (set correct type to field)
  3. templates (mappings)
  4. yeah, its module (regexp, patterns)

although, after reading more carefully, need to allocate time and answer them ... my answer is higher, in some points it is not quite correct.

  1. What @maxozerov said about this is correct. This is only possible with modules today. For instance, the apache module defines an ingest pipeline for parsing Apache HTTP server access logs here: https://github.com/elastic/beats/tree/master/filebeat/module/apache/access/ingest.

    To offer this as a "top-level" or "global" feature in Filebeat might be useful. To be honest, I'm not entirely sure why this is not the case already. If you have a GitHub account, I'd suggest opening an enhancement request for this feature here: https://github.com/elastic/beats/issues/new?template=feature-request.md. That way you can elaborate on your use case and follow along with the discussion and progress on the issue. If you don't have a GitHub account, let me know and I'll make an issue for you.

  2. @timestamp is a fundamental and, in fact, required field in all Beats events. There is no way to override it. Given that, perhaps you could just copy the value of myfield to the @timestamp field? Perhaps https://www.elastic.co/guide/en/elasticsearch/reference/master/date-processor.html could help?

  3. See 2.

  4. Perhaps https://www.elastic.co/guide/en/elasticsearch/reference/master/split-processor.html could help?

Cheers,

Shaunak

Hi @maxozerov, @shaunak,

Thanks for the replies.

  1. I thought the same, then we need an improvement for this, it would be great, there are multiple use cases.
    2, 3 and 4. Yes, processors and more processors, so the question here is: Is there a way to have a processor in Elasticsearch without doing a curl to enable it?

Regards