Filebeat in elastic agent is consuming more RAM in windows

Elastic Stack Elastic Agent
1

I am using elastic agent to collect custom logs from my windows server. Filebeat stream is consuming more than 1GB of RAM, some time it is consuming 3 to 4 GB RAM.
When i remove the filebeat configurations from the elastic-agent.yml file and collect only the system metrics, the RAM consumption is only 200 MB.

Configuration used to collect custom log data:

  - type: filestream
    id: filestream-data
    streams:
      - id: powershell-logs
        enabled: true
        paths:
          - C:\Transcripts\*\*.txt
        data_stream:
          type: logs         
          dataset: filebeat
          namespace: powershell       

        parsers:
        - multiline:
            type: pattern
            pattern: '^Command start time:'
            negate: true
            match: after  
        fields:
          log_type: powershell-logs

Can any one help why this is taking more RAM

Elastic agent version : 8.14
Elasticsearch version: 8.14
OS: Windows

image
image1347×476 71.3 KB

2

Your input is reading text files and using the multiline parser, depending on the size of those files this would be expected.

It needs to keep the lines in memory until a new match is found, so this will use memory, how much will probably depend on the number of files and the size of them.

3

This folder has 2 to 3 years of logs, How can i make it to send only the new logs and not to send old logs.