# Howto ingest from application logs and modules?

**URL:** <https://discuss.elastic.co/t/howto-ingest-from-application-logs-and-modules/272627>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [May 11, 2021, 3:21am UTC](https://discuss.elastic.co/t/howto-ingest-from-application-logs-and-modules/272627 "2021-05-11T03:21:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lachlan.simpson](https://avatars.discourse-cdn.com/v4/letter/l/50afbb/32.png) [@lachlan.simpson](https://discuss.elastic.co/u/lachlan.simpson)\
**Post date:** [May 11, 2021, 3:21am UTC](https://discuss.elastic.co/t/howto-ingest-from-application-logs-and-modules/272627/1 "2021-05-11T03:21:40Z")

</div>

We have a system from which we would like to collect the regular system modules.  
`filebeat modules enable system` and done. Great. Nothing is out of place.

But we have another application that requires an independent entry in **filebeat.yml**

```auto
filebeat.inputs:
- type: log
    enabled: true
    paths:
      - /var/spool/pbs/server_priv/accounting/*
    exclude_files: ['\.gz$']
    fields:
      pbs_accounting: true
    fields_under_root: true       

```

The output is set up for `logstash` in order to reap those non modularised logs.

But then I read that [filebeat modules work better reporting directly to elasticsearch](https://discuss.elastic.co/t/filebeat-system-module/150283/2) (admittedly an old post).

Then I read about [setting the indices specifically in filebeat to take advantage of the kibana dashboards](https://discuss.elastic.co/t/custom-index-name-for-filebeat-system-module-output/161892) (again, an old post).

Both of these seem like good things to have - but the documentation specifically states that [only a single output may be defined](https://www.elastic.co/guide/en/beats/filebeat/7.x/configuring-output.html).

So now I'm confused.

My first question is can modules be used with non-modularised app's logs defined in `filebeat.yml`?

If they can't, then presumably I need to write the parser for the `system logs` in logstash? Something like this? Or are there more complex matches available? this is where we are swinging around to the real qy

```auto
filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } 
  }
  date {
    match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]   
  }

```

This is where we are swinging around to the real question - I would love access to [the fields listed as parsed in the system module](https://www.elastic.co/guide/en/beats/filebeat/7.x/exported-fields-system.html) but that simple match above isn't going to get me there.

Is there a way I can get the rich field set of the `filebeat system module` while still harvesting a non regular application's logs and sending them to logstash?

---

<div class="post-metadata">

**Author:** ![lachlan.simpson](https://avatars.discourse-cdn.com/v4/letter/l/50afbb/32.png) [@lachlan.simpson](https://discuss.elastic.co/u/lachlan.simpson)\
**Post date:** [May 11, 2021, 3:42am UTC](https://discuss.elastic.co/t/howto-ingest-from-application-logs-and-modules/272627/2 "2021-05-11T03:42:24Z")

</div>

Oooooh. _This_ is how I get the rich system parsing in logstash: [use Logstash pipelines for parsing](https://www.elastic.co/guide/en/logstash/7.x/logstash-config-for-filebeat-modules.html#parsing-system)? Does it mean I need to set

```auto
fields:
  fileset.module => "system"

```

in order to get the parsing write in that filter?

Is there a way to set up `filebeat` so that modules go to a different port so I can utilize [multiple pipeline config](https://www.elastic.co/blog/logstash-multiple-pipelines)? IE, send filebeat defined modules to `port 5045` and other logs to `port 5044`?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2021, 5:42am UTC](https://discuss.elastic.co/t/howto-ingest-from-application-logs-and-modules/272627/3 "2021-06-08T05:42:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.