How can I Filebeat multiple files that are radically different

Elastic Stack Beats
1

Is this community still active ?

I want to ingest two files from my web server, the apache access logs and the modsecurity logs.

The modsecurity logs I will have to setup some serious GROK filters.

Filebeat and my index pattern works fine for the access logs but when I add the modsecurity logs no logs ingest. (Modsec_audit.logs are far more random not delineated by LF CR or characters.)

How do people handle this?

I tried setting up a tag for each log and adding the tag to the index to split them into separate indexes, but was not successful.

Can I send some logs in my filebeat config to Elastic and some to Logstash ?

my enabled apache module yaml:
/etc/filebeat/modules.d/apache2.yml

- module: apache2
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    var.paths: ['/var/log/httpd/zones/ssl_access_log']
    fields:
      type: "access"
    var.paths: ['/var/log/httpd/zones/modsec_audit.log']
    fields:
      type: "modsec"

  # Error logs
  error:
    enabled: true

filebeat.yml

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.0.0.5:9200"]
  index: "weblog%{[fields.type]}%"
#ilm.enabled: false
#setup.ilm.enabled: false


setup.template.name: "weblog"
setup.template.pattern: "weblog*"

filebeat test config
filebeat test output

both pass, but no new indices are created.
Tried disabling ILM and leaving default as above.

2

How are you setting the input of the modsecurity logs?

Why do you ask that?

3

Blockquote
How are you setting the input of the modsecurity logs?

serial logging so are captured in a single file.

SecDebugLogLevel 0
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial

separate team members manage this and cannot adjust although might be able to request an additional log.

Perhaps I can setup some GROK filters if it was going to a second index that didn't disrupt my ssl_access log patterns.

Blockquote
Why do you ask that?

Been to a few forums where the bulk of the community had moved to a new venue.

I wonder if sending Logstash would be better suited for GROK out certain strings like:
Host, User-Agent, Referer,X-Forwarded-For.
Current Apache version cannot get X-Forwarded-For (which is clientIP because of Load balancer)

Can I have filebeat send 1 file to ELK and 1 file to logstash to separate indicies ?

4

Any idea on how we can send different log files to different indices?

5

I meant in Filebeat.

This has always been the core community forum for Elasticsearch, so I can't comment on any others that you may have been to.

6

and that is why I ask.
Any suggestions? Anyone using ELK for more than one file with filebeat ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.