Filebeat processor for single line logs?

Elastic Stack Beats
1

I ran into this threat in Github so I just copy-paste:

  • Steps to Reproduce: Enable Apache module in Filebeat and use the DNS processor; configuration example below:
- dns:
    type: reverse
    action: append
    fields:
      source.ip: source.hostname
      destination.ip: destination.hostname
    success_cache:
      capacity.initial: 1000
      capacity.max: 10000
    failure_cache:
      capacity.initial: 1000
      capacity.max: 10000
      ttl: 1m
    nameservers: ['8.8.8.8', '8.8.4.4']
    timeout: 500ms
    tag_on_failure: [_dns_reverse_lookup_failed]

In my particular case I'm also using other processors, namely the add_fields processor. While that processor works, DNS silently fails.

There will be no source.hostname or destination.hostname in the http events dispatched to Elasticseach.

On a second thought I've noticed that this is not working for the auth module either, which leads me to believe that the DNS processor will not work on single line logs like Apache and SSH.

It's possible that this is related to the order in which the processor is ran. If it happens after the Filebeat pipeline transforms that single line log to ECS-mapped event, it should work (as the fields configured in the DNS processor are present), but if it happens before then it's not expected to work, as those single line logs don't hold any field:value schema.

2

Did you try to enable debug logging?

3

No, mostly because I wanted to understand if what i want to do is possible or no. is filebeat able to use the dns processor for Apache, for exanple?

If so and not working i can troubleshot more.

4

Is filebeat able to use the dns processor for Apache, for example?

yes, it's available and described here: DNS Reverse Lookup | Filebeat Reference [master] | Elastic I don't expect any interference between processors. Please recheck your configuration.

5

I even copy paste the configuration above. it doesn't work. you can try for yourself. use dns processor and enable apache module. it will not work. try ssh module. it will not work.

processor doesnt pick none of them. if you try suricata. works.

6

Is ther docs that explain order of processors?

is

source -> ecs -> processor?
source -> processor -> ecs?

processor doesnt work on modules apache and ssh and docs dont help.

7

Unfortunately, you cannot use the DNS processor like this with the Apache module. The fields source.ip and destination.ip are extracted from the log lines on Elasticsearch. Thus, when Filebeat tries to run the DNS processor, it cannot do anything, as those fields do not exist at that point of processing.

DNS processor can only work if you add the fields to the event beforehand.

processors:
- add_fields:
  source.ip: 1.2.3.4
- dns:
  your_dns_config
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.