Fatal error: runtime: cannot map pages in arena address space

Elastic Stack Beats
21

Hi Ruflin,

After it failed, I am running it using the command line with -e -d'*' options.
Used the windows performance monitor to show the handles count of the filebeat process.
Got Max number of 2,137
The process fails again with the message of:

fatal error: runtime: cannot map pages in arena address space

In another run it got to 3,317 handles

Ori

22

I'm not sure if your issue is related to the open files or the size of the registry file. In the 5.0 release we have config options to potentially fix both. Can you try the clean_ options to reduce the size of the registry? Also can you share your config file?

23

Hi,

I am currently testing Version 5.0
But on prod we are running with 1.2.3.
5.0 is currently Alpha, so will not put it prod.

Config File (Part1):

filebeat:
  registry_file: C:/ProgramData/filebeat/registry
  prospectors:
  #Access
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\Access\Access*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: access
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 310
        time_to_keep: 43200

  #AccessControlLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\AccessControlLogs\accesscontrol*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: accesscontrollogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 311
        time_to_keep: 43200

  #actionAudit
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\actionAudit\temp\audit*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: actionaudit
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 113
        time_to_keep: 43200

  #AdminLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\AdminLogs\admin*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: adminlogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 410
        time_to_keep: 43200

  #AppLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\AppLogs\RT*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: applogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 110
        time_to_keep: 43200

  #BlazedsLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\BlazedsLogs\blazeds*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: blazeds
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 956
        time_to_keep: 43200

  #BlazedsTraceLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\BlazedsTraceLogs\blazedsTrace*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: blazedstrace
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 211
        time_to_keep: 43200

  #GoProAuditLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\GoProAuditLogs\goproaudit_RT*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: gopro
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 950
        time_to_keep: 43200

  #integrationsLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\integrationsLogs\integrations*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: integrationlogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 953
        time_to_keep: 43200

  #KontikiLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\KontikiLogs\audit*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: kontiki
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 111
        time_to_keep: 43200

  #MicrositeHitsLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\MicrositeHitsLogs\micrositehits_RT*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: micrositehits
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 954
        time_to_keep: 43200

Ori

24

Config File Part 2:

  #profiler
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\profiler\profiler*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: profiler
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 212
        time_to_keep: 43200

  #reportingLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\reportingLogs\reporting*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: reportinglogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 952
        time_to_keep: 43200

  #SamlLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\SamlLogs\temp\audit*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: samllogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 951
        time_to_keep: 43200

  #SecurityLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\SecurityLogs\security*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: securitylogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 312
        time_to_keep: 43200

  #UserExperienceLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\UserExperienceLogs\UserExperience_RT*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: userexperiencelogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 955
        time_to_keep: 43200

  #VCULogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\VCULogs\audit*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: vculogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 114
        time_to_keep: 43200

  #WLS
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\WLS\RT*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: wls
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 210
        time_to_keep: 43200

  #SOAPLogs
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\Company\servers\SRV1P\Logs\SOAPLogs\audit*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: soaplogs
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 810
        time_to_keep: 43200

  #Api
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\API\servers\SRV1-API\logs\consoleAPILogs\api*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: api
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 510
        time_to_keep: 43200

  #Uci
    -
      paths:
        - D:\Bea12\oracle\middleware\domains\UCI\servers\SRV1-UCI\logs\RT*.log*
      input_type: log
      ignore_older: 2h
      close_older: 5m
      document_type: uci
      harvester_buffer_size: 16384
      encoding: plain
      fields_under_root: true
      fields:
        app_env: Prod
        type_id: 610
        time_to_keep: 43200

    # Heartbeat
    -
      paths:
        - c:\ProgramData\Filebeat\Logs\heartbeat.log.*
      input_type: log
      ignore_older: 3m
      close_older: 1m
      document_type: heartbeat
      harvester_buffer_size: 16384
      encoding: plain

output:
  file:
    path: c:/ProgramData/Filebeat/Logs
    filename: filebeatdata
    rotate_every_kb: 20480
    number_of_files: 10
  logstash:
    hosts: ["MyELKServer:5101"]
    max_retries: -1
25

Hi Ori

I'm sorry I never replied to this thread here. Seems like I missed it somehow. You are probably happy to hear that we just merged yesterday a PR that allows to limit the max number of harvesters per Prospector: https://github.com/elastic/beats/pull/2417 This could help in your case.

26

Thanks Ruflin,

On which version of filebeat will it be available ?

Ori

27

The first version this will be available is 5.0.0-beta1