# How can I Filebeat multiple files that are radically different

**URL:** <https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 13, 2022, 11:56pm UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350 "2022-09-13T23:56:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dimmthewitted1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimmthewitted1/32/110086_2.png) [@dimmthewitted1](https://discuss.elastic.co/u/dimmthewitted1)\
**Post date:** [September 13, 2022, 11:56pm UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/1 "2022-09-13T23:56:30Z")

</div>

Is this community still active ?

I want to ingest two files from my web server, the apache access logs and the modsecurity logs.

The modsecurity logs I will have to setup some serious GROK filters.

Filebeat and my index pattern works fine for the access logs but when I add the modsecurity logs no logs ingest. (Modsec\_audit.logs are far more random not delineated by LF CR or characters.)

How do people handle this?

I tried setting up a tag for each log and adding the tag to the index to split them into separate indexes, but was not successful.

Can I send some logs in my filebeat config to Elastic and some to Logstash ?

my enabled apache module yaml:  
/etc/filebeat/modules.d/apache2.yml

```auto
- module: apache2
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    var.paths: ['/var/log/httpd/zones/ssl_access_log']
    fields:
      type: "access"
    var.paths: ['/var/log/httpd/zones/modsec_audit.log']
    fields:
      type: "modsec"

  # Error logs
  error:
    enabled: true

```

filebeat.yml

```auto
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.0.0.5:9200"]
  index: "weblog%{[fields.type]}%"
#ilm.enabled: false
#setup.ilm.enabled: false

setup.template.name: "weblog"
setup.template.pattern: "weblog*"

```

filebeat test config  
filebeat test output

both pass, but no new indices are created.  
Tried disabling ILM and leaving default as above.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 14, 2022, 6:25am UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/2 "2022-09-14T06:25:36Z")

</div>

How are you setting the input of the modsecurity logs?

> [@dimmthewitted1](#):
>
> Is this community still active ?

Why do you ask that?

---

<div class="post-metadata">

**Author:** ![dimmthewitted1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimmthewitted1/32/110086_2.png) [@dimmthewitted1](https://discuss.elastic.co/u/dimmthewitted1)\
**Post date:** [September 15, 2022, 3:43pm UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/3 "2022-09-15T15:43:47Z")

</div>

> Blockquote  
> How are you setting the input of the modsecurity logs?

serial logging so are captured in a single file.

```
SecDebugLogLevel 0
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial

```

separate team members manage this and cannot adjust although might be able to request an additional log.

Perhaps I can setup some GROK filters if it was going to a second index that didn't disrupt my ssl\_access log patterns.

> Blockquote  
> Why do you ask that?

Been to a few forums where the bulk of the community had moved to a new venue.

I wonder if sending Logstash would be better suited for GROK out certain strings like:  
Host, User-Agent, Referer,X-Forwarded-For.  
Current Apache version cannot get X-Forwarded-For (which is clientIP because of Load balancer)

Can I have filebeat send 1 file to ELK and 1 file to logstash to separate indicies ?

---

<div class="post-metadata">

**Author:** ![dimmthewitted1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimmthewitted1/32/110086_2.png) [@dimmthewitted1](https://discuss.elastic.co/u/dimmthewitted1)\
**Post date:** [September 19, 2022, 11:43pm UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/4 "2022-09-19T23:43:36Z")

</div>

Any idea on how we can send different log files to different indices?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 19, 2022, 11:46pm UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/5 "2022-09-19T23:46:36Z")

</div>

> [@dimmthewitted1](#):
>
> > How are you setting the input of the modsecurity logs?

I meant in Filebeat.

> [@dimmthewitted1](#):
>
> Been to a few forums where the bulk of the community had moved to a new venue.

This has always been the core community forum for Elasticsearch, so I can't comment on any others that you may have been to.

---

<div class="post-metadata">

**Author:** ![dimmthewitted1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimmthewitted1/32/110086_2.png) [@dimmthewitted1](https://discuss.elastic.co/u/dimmthewitted1)\
**Post date:** [September 20, 2022, 10:43pm UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/6 "2022-09-20T22:43:45Z")

</div>

> [@dimmthewitted1](#):
>
> Is this community still active ?

and that is why I ask.  
Any suggestions? Anyone using ELK for more than one file with filebeat ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2022, 12:44am UTC](https://discuss.elastic.co/t/how-can-i-filebeat-multiple-files-that-are-radically-different/314350/7 "2022-10-19T00:44:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.