Ignore_inactive does not work in filebeat with filestream config type

Elastic Stack Beats
1

When I use the filestream type instead of the log type, filebeat always reads the entire log file from the beginning.

- type: filestream
  id: test_id
  enable: true
  paths:
    - "/usr/share/filebeat/inputs.d/*.log"
  parsers:
    - multiline:
        type: pattern
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after
  ignore_inactive: since_first_start

No matter of using since_first_start or since_last_start, it doesn't work.

But when I use log config type with tail_files:true, it works normally.

- type: log
  paths:
    - "/usr/share/filebeat/inputs.d/*.log"
  multiline:
    type: pattern
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
  enable: true
  tail_files: true

So my question is, how can I configure the filestream type so that filebeat doesn't read my log file from scratch?

Thanks to everyone who can solve it .

2

That's when you restart Filebeat or what is the trigger for that?

3

The problem I have detected is the following:

When I use ignore_inactive: since_first_start: Reads the updated file lines from startup (does not read the whole file), however, when Filebeat is restarted, it reads the whole file, duplicating the lines.

When I use ignore_inactive: since_last_start: Reads the lines of the file updated since startup (does not read the whole file), however, when Filebeat is restarted or stopped for a while, it reads the lines again since startup, losing lines.

Thanks.

4

@xeraa Do you have a record of what I am telling you, or is it a bug?

5

Quick question, is this a network share?

6

@leandrojmp :
No, it is an example, if you configure a file in /home/user/*.log it has the same behavior.

7

when Filebeat is restarted, it reads the whole file, duplicating the lines.

when Filebeat is restarted or stopped for a while, it reads the lines again since startup, losing lines.

It kind of sounds like you don't keep the state between restarts? What's in your registry file and can that be written to and read by Filebeat (and persists between restarts)? How Filebeat works | Filebeat Reference [8.11] | Elastic for how the state is supposed to work.

I kind of doubt that this is generally broken (otherwise all hell would have broken lose) but maybe there's something in your environment that causes an issue there?

8

Hi @xeraa :

Thx for reply, the configuration we have is:


# ============================== Filebeat inputs ===============================
filebeat.inputs:
###### INPUT 1 #####
- type: filestream
  id: bbdd_mw_cpd
  enabled: true
  paths:
     - /opt/oraas/logs/as/wls/*/*access.log
  tags: ["beats", "bbdd_mw_cpd"]

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

# ------------------------------ Logstash Output -------------------------------

output.logstash:
   hosts: ["logstash.ocp02.cm1.mercadona.com:4003"]


logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

The registry files, have the following permissions:

Path: /opt/filebeat/data/registry/filebeat

image
image711×101 3.27 KB

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.