How can I Filebeat multiple files that are radically different

Is this community still active ?

I want to ingest two files from my web server, the apache access logs and the modsecurity logs.

The modsecurity logs I will have to setup some serious GROK filters.

Filebeat and my index pattern works fine for the access logs but when I add the modsecurity logs no logs ingest. (Modsec_audit.logs are far more random not delineated by LF CR or characters.)

How do people handle this?

I tried setting up a tag for each log and adding the tag to the index to split them into separate indexes, but was not successful.

Can I send some logs in my filebeat config to Elastic and some to Logstash ?

my enabled apache module yaml:

- module: apache2
  # Access logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ['/var/log/httpd/zones/ssl_access_log']
      type: "access"
    var.paths: ['/var/log/httpd/zones/modsec_audit.log']
      type: "modsec"

  # Error logs
    enabled: true


  # Array of hosts to connect to.
  hosts: [""]
  index: "weblog%{[fields.type]}%"
#ilm.enabled: false
#setup.ilm.enabled: false "weblog"
setup.template.pattern: "weblog*"

filebeat test config
filebeat test output

both pass, but no new indices are created.
Tried disabling ILM and leaving default as above.

How are you setting the input of the modsecurity logs?

Why do you ask that?

How are you setting the input of the modsecurity logs?

serial logging so are captured in a single file.

SecDebugLogLevel 0
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial

separate team members manage this and cannot adjust although might be able to request an additional log.

Perhaps I can setup some GROK filters if it was going to a second index that didn't disrupt my ssl_access log patterns.

Why do you ask that?

Been to a few forums where the bulk of the community had moved to a new venue.

I wonder if sending Logstash would be better suited for GROK out certain strings like:
Host, User-Agent, Referer,X-Forwarded-For.
Current Apache version cannot get X-Forwarded-For (which is clientIP because of Load balancer)

Can I have filebeat send 1 file to ELK and 1 file to logstash to separate indicies ?

Any idea on how we can send different log files to different indices?

I meant in Filebeat.

This has always been the core community forum for Elasticsearch, so I can't comment on any others that you may have been to.

and that is why I ask.
Any suggestions? Anyone using ELK for more than one file with filebeat ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.