# Logstash Pipelines for Parsing Questions - No fileset in output **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 17, 2020, 1:11pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519 "2020-04-17T13:11:40Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [April 17, 2020, 1:11pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/1 "2020-04-17T13:11:41Z")
I've tried following [https://www.elastic.co/guide/en/logstash/7.6/logstash-config-for-filebeat-modules.html](https://www.elastic.co/guide/en/logstash/7.6/logstash-config-for-filebeat-modules.html), and create logstash pipelines to parse the filebeat data. However, even the examples provided in that link don't seem to apply to me. The example config begins with filter { if [fileset][module] == "system" { if [fileset][name] == "auth" { Well, when I look at my filebeat logs and just dump them direct to stdout or a file, there is no "fileset.module", "fileset.name" or even "fileset" anywhere in the logs, so their example parsing config never matches. I don't understand if their recommended config is wrong or if there is something I need to do still to get my filebeat output to have "fileset" values. On the filebeat side, I have the system module enabled and the output going to logstash (on a custom port not 5044). Other than that, it is the default install of filebeat. Do you have any ideas on this? Thanks for any help you can give. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 17, 2020, 5:48pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/2 "2020-04-17T17:48:02Z")
Hi @mgotechlock, could you post here a couple of the filebeat events you dumped to stdout/file? Please make sure to redact any sensitive information in the events before posting. Thanks, Shaunak ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [April 17, 2020, 6:11pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/3 "2020-04-17T18:11:34Z")
```auto { "input" => { "type" => "log" }, "cloud" => { "instance" => { "id" => "156963326" }, "provider" => "digitalocean", "region" => "nyc3" }, "@version" => "1", "@timestamp" => 2020-04-17T18:08:02.238Z, "host" => { "name" => "oompaloompa", "architecture" => "x86_64", "hostname" => "oompaloompa", "os" => { "name" => "Ubuntu", "codename" => "bionic", "platform" => "ubuntu", "family" => "debian", "kernel" => "4.15.0-96-generic", "version" => "18.04.4 LTS (Bionic Beaver)" }, "containerized" => false, "id" => "fc61c6cc61c1434fbf7d14b4fbff55f6" }, "ecs" => { "version" => "1.4.0" }, "tags" => [ [0] "beats_input_codec_plain_applied" ], "message" => "Apr 14 21:24:08 oompaloompa sshd[6934]: Invalid user redis1 from 203.159.249.215 port 58702", "agent" => { "ephemeral_id" => "d9f4cc34-072e-4b08-bd19-247c9114c9e5", "version" => "7.6.2", "type" => "filebeat", "hostname" => "oompaloompa", "id" => "469f2965-149e-4064-9e0c-2cd0669728b5" }, "log" => { "file" => { "path" => "/var/log/auth.log" }, "offset" => 943508 } } ``` ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [April 17, 2020, 6:12pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/4 "2020-04-17T18:12:25Z")
```auto "input" => { "type" => "log" }, "cloud" => { "instance" => { "id" => "156963326" }, "region" => "nyc3", "provider" => "digitalocean" }, "@version" => "1", "@timestamp" => 2020-04-17T18:08:02.238Z, "host" => { "name" => "oompaloompa", "architecture" => "x86_64", "hostname" => "oompaloompa", "os" => { "name" => "Ubuntu", "codename" => "bionic", "family" => "debian", "platform" => "ubuntu", "kernel" => "4.15.0-96-generic", "version" => "18.04.4 LTS (Bionic Beaver)" }, "containerized" => false, "id" => "fc61c6cc61c1434fbf7d14b4fbff55f6" }, "ecs" => { "version" => "1.4.0" }, "tags" => [ [0] "beats_input_codec_plain_applied" ], "message" => "Apr 14 21:24:08 oompaloompa sshd[6934]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=203.159.249.215", "agent" => { "ephemeral_id" => "d9f4cc34-072e-4b08-bd19-247c9114c9e5", "version" => "7.6.2", "type" => "filebeat", "hostname" => "oompaloompa", "id" => "469f2965-149e-4064-9e0c-2cd0669728b5" }, "log" => { "offset" => 943686, "file" => { "path" => "/var/log/auth.log" } } } ``` ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 17, 2020, 6:34pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/5 "2020-04-17T18:34:47Z")
Hmm, that's strange. I just tried to reproduce this with Filebeat 7.6.0 with the `system` module enabled and I'm seeing an `event` field in the events, which contains sub-fields like `module` and `dataset`. Could you please post the result of the following command (again, after redacting any sensitive information)? ```auto filebeat export config ``` Thanks, Shaunak ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [April 17, 2020, 11:22pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/6 "2020-04-17T23:22:25Z")
So I made progress. Apparently, you are not supposed to have inputs enabled in filebeat.yml AND the modules enabled. Who knew? The documentation is terrible. Once I disabled the inputs from filebeat.yml, I see data in a better format, but still insufficient to meet the config Elastic publishes in the original URL. I do see fileset.name=auth and event.module=system, but the example config is fileset.module=system. I can easily change the config but i would like confirmation that the example config Elastic publishes is incorrect, so I can be sure I am not doing anything wrong. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 17, 2020, 11:58pm UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/7 "2020-04-17T23:58:03Z")
> [@mgotechlock](#): > > Apparently, you are not supposed to have inputs enabled in filebeat.yml AND the modules enabled. It's possible to mix "raw" Filebeat inputs in your configuration with modules. Imagine a case where you have logs from a well-known service like Apache or system logs but also have logs from your own application. You could ingest all of these logs with a single Filebeat instance by enabling the apache and system modules but also specifying your own "raw" inputs in the Filebeat configuration. BTW, modules start their own inputs under the hood so, at the end of the day, there are inputs configured anyway! > [@mgotechlock](#): > > I do see fileset.name=auth and event.module=system, but the example config is fileset.module=system. I can easily change the config but i would like confirmation that the example config Elastic publishes is incorrect, so I can be sure I am not doing anything wrong. You're right — the configurations shown in that documentation are outdated. So sorry about that and thank you for bringing it to our attention. I've created a PR now to fix that documentation: [Updating fields to new ECS names by ycombinator · Pull Request #11807 · elastic/logstash · GitHub](https://github.com/elastic/logstash/pull/11807). As I've done in the PR, I'd suggest using `[event][module]` in place of `[fileset][module]` and `[event][dataset]` instead of `[fileset][name]`. Note that the `event.dataset` field contains the **fully-qualified** name of the dataset (aka fileset), so it includes the module name as a prefix, e.g. `system.auth`. ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [April 20, 2020, 11:13am UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/8 "2020-04-20T11:13:06Z")
Awesome. Thanks for your assist. I understand it now. If i could ask one more question. Just taking the [auth] from system pipeline, I've enabled it and are getting logs but some are not being parsed properly, while some are. I believe it is because the unparsed ones do not match one of the 7 "match" statements in the example config. I can obviously add more but I just wanted confirmation that that is to be expected and that I should not expect the example config to catch every possible entry? And if that is true, any idea how many "match" statements I might end up needing to create ? ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [April 20, 2020, 11:30am UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/9 "2020-04-20T11:30:28Z")
FYI, if you do change the example to event.dataset, change the value to system.auth. My logs are still showing fileset.name = auth works, so I think either is fine, though I admittedly only looking at ubuntu at the moment. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 20, 2020, 11:45am UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/10 "2020-04-20T11:45:47Z")
> [@mgotechlock](#): > > If i could ask one more question. > Just taking the [auth] from system pipeline, I've enabled it and are getting logs but some are not being parsed properly, while some are. I believe it is because the unparsed ones do not match one of the 7 "match" statements in the example config. I can obviously add more but I just wanted confirmation that that is to be expected and that I should not expect the example config to catch every possible entry? And if that is true, any idea how many "match" statements I might end up needing to create ? Would you mind creating a new topic for this, since this topic here is already marked as solved? It just keeps the forums clean and easily searchable for anyone else running into similar problems. Thanks! ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 20, 2020, 11:48am UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/11 "2020-04-20T11:48:43Z")
> [@mgotechlock](#): > > FYI, if you do change the example to event.dataset, change the value to system.auth. Indeed, that's what I meant by: > [@shaunak](#): > > Note that the `event.dataset` field contains the **fully-qualified** name of the dataset (aka fileset), so it includes the module name as a prefix, e.g. `system.auth` . 🙂 > [@mgotechlock](#): > > My logs are still showing fileset.name = auth works, so I think either is fine Yes definitely, either can work, at least for now. The reason I prefer to use `event.dataset` is that it's a [core field in ECS](https://www.elastic.co/guide/en/ecs/current/ecs-event.html) and, as such, more future proof. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 18, 2020, 11:48am UTC](https://discuss.elastic.co/t/logstash-pipelines-for-parsing-questions-no-fileset-in-output/228519/12 "2020-05-18T11:48:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.