# Usage of filestream

**URL:** <https://discuss.elastic.co/t/usage-of-filestream/320009>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 29, 2022, 8:07am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009 "2022-11-29T08:07:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [November 29, 2022, 8:07am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/1 "2022-11-29T08:07:23Z")

</div>

Hello,

I have a few question about the topic filestream and it's difference to the input log.

1. Do I only need an id when using multiple filebeat inputs in a single yml or always? Currently im not using any ids but im curious if i may run into trouble.

2. Can prospector options be nested e.g. like:

```auto
prospector
  scanner
    exclude_files: ...
    check_interval: ...

```

1. How does the `exclude_files` work? In the migrating to filestream docu ([Step 2: Exclude all processed files | Filebeat Reference [8.5] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/_step_2_exclude_all_processed_files.html)) they got:

```auto
  paths:
    - /var/log/my-application*.json
  prospector.scanner.exclude_files: my-application[1-2]{1}.log

```

Does this mean that my-application\*.log is excluded from the path /var/log/ or where is the exclusion happening?

1. How are multiple excluded files separeted? i'd assume its ['file1\_pattern', 'file2\_pattern']?

2. I'm using scan\_frequency with type filestream, so according to the renaming table ([Step 3: Use new option names | Filebeat Reference [8.5] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/_step_3_use_new_option_names.html)), it's not working. do both options still work anyways or do i have to rename it in every .yml?

3. What is the difference between e.g. paths: /var/log/_.log and include\_files: /var/log/_.log? When would i use one over the other?

Thanks in advance,  
Ossenfeld

---

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [November 30, 2022, 9:18am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/2 "2022-11-30T09:18:51Z")

</div>

Any one on question 1 at least? 🙂

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [December 1, 2022, 11:18pm UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/3 "2022-12-01T23:18:24Z")

</div>

@faec Could you help with these questions please? Thanks!!

---

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [December 5, 2022, 9:18am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/4 "2022-12-05T09:18:40Z")

</div>

Might it help if i open a thread for every question?

---

<div class="post-metadata">

**Author:** ![faec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faec/32/46988_2.png) [@faec](https://discuss.elastic.co/u/faec)\
**Post date:** [December 6, 2022, 10:08pm UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/5 "2022-12-06T22:08:45Z")

</div>

1. We recommend always using an id, but current releases will assign a default for single inputs.
2. Yes but `prospector` and `scanner` should be followed by a colon `:`
3. In filestream this parameter is a list so you should instead use `exclude_files: ["my-application[1-2]{1}.log", "some-other-pattern..."]`. The exclusions are regular expressions, and any file that matches that regular expression will not be ingested.
4. (see previous example)
5. No, if you switch to the filestream input then any instances of the old `scan_frequency` parameter should be replaced with `prospector.scanner.check_interval`
6. `paths` specifies where the input should look for possible files. If you want to ingest all files matching those paths, then there's no need to do anything else. If you want to only ingest _some_ of those files, then adding a regular expression to `include_files` will only ingest files that are in one of the configured paths _and_ match the given regular expression.

---

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [December 9, 2022, 10:04am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/6 "2022-12-09T10:04:20Z")

</div>

Exclude\_files seems to be buggy I think. I used the two following filebeat configurations:

```auto
# vim: ft=yaml

- type: filestream
  id: json-collector
  paths:
  ¦ - /var/log/parser-testing/*
  fields:
  ¦ parser.test: "json_only"

  fields_under_root: true
  ignore_older: 30m
  close.on_state_change.inactive: 5m
  prospector:
  ¦ scanner:
  ¦ ¦ check_interval: 1s
  ¦ ¦ exclude_files: [".*.log"]
  parsers:
  ¦ - ndjson:
  ¦ ¦ ¦ keys_under_root: true
  ¦ ¦ ¦ expand_keys: true
  ¦ ¦ ¦ add_error_key: true

```

and

```auto
---
# vim: ft=yaml

- type: filestream
  id: log-collector
  paths:
  ¦ - /var/log/parser-testing/*
  fields:
  ¦ parser.test: "json_excluded"

  fields_under_root: true
  ignore_older: 30m
  close.on_state_change.inactive: 5m
  prospector:
  ¦ scanner:
  ¦ ¦ check_interval: 1s
  ¦ ¦ exclude_files: [".*.json"]
  parsers:
  ¦ - multiline:
  ¦ ¦ ¦ type: pattern
  ¦ ¦ ¦ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  ¦ ¦ ¦ match: after
  ¦ ¦ ¦ negate: true

```

Files named xyz.log are collected, but xyz.json isn't. When removing the exclude\_files, everything including xyz.json is collected. I could just specify the path like \*.log and \*.json, but I really would like to know what's going on with the exclude\_files?

---

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [December 15, 2022, 7:00am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/7 "2022-12-15T07:00:27Z")

</div>

Any ideas?

---

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [January 4, 2023, 6:24am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/8 "2023-01-04T06:24:54Z")

</div>

Maybe one last bump 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 1, 2023, 8:25am UTC](https://discuss.elastic.co/t/usage-of-filestream/320009/9 "2023-02-01T08:25:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.