Unable to ingest apache2 logs through filebeat

Hi Team,
I am trying to ingest the apache 2 access and error logs through the filebeat index and getting the below error

./filebeat -e -c filebeat.yml -d "publish"
2018-08-28T02:40:42.896-0700    INFO    instance/beat.go:468    Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-08-28T02:40:42.896-0700    INFO    instance/beat.go:475    Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5
2018-08-28T02:40:42.896-0700    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.4
2018-08-28T02:40:42.896-0700    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-08-28T02:40:42.896-0700    INFO    pipeline/module.go:76   Beat name: STOVLNX3124.corp.frk.com
2018-08-28T02:40:42.899-0700    INFO    beater/filebeat.go:62   Enabled modules/filesets: apache2 (access, error),  ()
2018-08-28T02:40:42.899-0700    ERROR   instance/beat.go:667    Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log
Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log

and my filebeat.yml file look like below

#==========================  Modules configuration ============================
filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: /opt/bea/Logwarhouse/MarketApache/*access.log


  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: /opt/bea/Logwarhouse/MarketApache/*error.log


#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/modules.d/apache2.yml

  # Set to true to enable config reloading
  #reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

  #setup.template.settings:
  #index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "elastic"

In addition to the above setup, i executed the below command as well and also installed ingest-user-agent and ingest-geoip..

 ./filebeat -e -M "apache2.access.var.paths=[/opt/bea/Logwarhouse/MarketApache/*access.log]" -M "apache2.error.var.paths=[/opt/bea/Logwarhouse/MarketApache/*error.log]"
2018-08-28T05:50:50.573-0700    INFO    instance/beat.go:468    Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-08-28T05:50:50.574-0700    INFO    instance/beat.go:475    Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5
2018-08-28T05:50:50.574-0700    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.4
2018-08-28T05:50:50.574-0700    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-08-28T05:50:50.575-0700    INFO    pipeline/module.go:76   Beat name: STOVLNX3124.corp.frk.com
2018-08-28T05:50:50.578-0700    INFO    beater/filebeat.go:62   Enabled modules/filesets: apache2 (access, error),  ()
2018-08-28T05:50:50.579-0700    INFO    instance/beat.go:301    filebeat start running.
2018-08-28T05:50:50.579-0700    INFO    registrar/registrar.go:110      Loading registrar data from /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data/registry
2018-08-28T05:50:50.579-0700    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-08-28T05:50:50.579-0700    INFO    registrar/registrar.go:121      States Loaded from registrar: 3
2018-08-28T05:50:50.579-0700    INFO    crawler/crawler.go:48   Loading Prospectors: 2
2018-08-28T05:50:50.580-0700    INFO    log/prospector.go:111   Configured paths: [/opt/bea/Logwarhouse/MarketApache/*access.log]
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/uk-access.log
2018-08-28T05:50:50.581-0700    INFO    log/prospector.go:111   Configured paths: [/opt/bea/Logwarhouse/MarketApache/*error.log]
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/us-dev-3-access.log
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-qa-access.log
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/ind-dev-1-access.log
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-dev4-access.log
2018-08-28T05:50:50.583-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/error.log*]
2018-08-28T05:50:50.584-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*]
2018-08-28T05:50:50.584-0700    INFO    crawler/crawler.go:82   Loading and starting Prospectors completed. Enabled prospectors: 2
2018-08-28T05:50:50.584-0700    INFO    cfgfile/reload.go:127   Config reloader started
2018-08-28T05:50:50.587-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*]
2018-08-28T05:50:50.588-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/error.log*]
2018-08-28T05:50:50.588-0700    INFO    cfgfile/reload.go:258   Starting 1 runners ...
2018-08-28T05:50:50.588-0700    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-08-28T05:50:50.591-0700    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.4
2018-08-28T05:50:50.592-0700    ERROR   fileset/factory.go:93   Error loading pipeline: Error loading pipeline for fileset apache2/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
    sudo bin/elasticsearch-plugin install ingest-user-agent
    sudo bin/elasticsearch-plugin install ingest-geoip

bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%  
-> Installed ingest-user-agent


 bin/elasticsearch-plugin install ingest-geoip
-> Downloading ingest-geoip from elastic
[=================================================] 100%  
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip

The vars.paths setting in the apache module requires an array. The template engine complaints about it being a plain string.

Update your modules setting to say:

filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - '/opt/bea/Logwarhouse/MarketApache/*access.log'


  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - '/opt/bea/Logwarhouse/MarketApache/*error.log'

Thank you Steffen, I am able to create filebeat index for apache access logs now, but although i have included the error log in above file, it was not picked up. May i know what would be the possible cause for this.

May i know what would be the possible cause for this.

You mean the cause for th error message Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log ?

Beats uses a template engine to expand user settings into a configuration object. The template requires var.paths to be a list of strings. You originally did not configure a list of string, but a string value only. Normally the list of strings is iterated, creating one configuration per entry. A single string can not be iterated. It's basically a type error.

No Not for this, I am saying apache error logs are getting picked up by the index.

Can you share filebeat logs? Have you checked that the path is correct?

Hi Steffens,
Let me park the above discussion. I have some additonal details captured in the apache access log and i would like to know if the filebeat can parse them as well.

Myapache access looks like below.

0 10.65.19.201 - - [22/May/2018:00:08:50 -0700] "GET /assets/css/print.css HTTP/1.1" 200 509 "https://roddoc.patqa.corp.rrk.com/profile-web/signin" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) li
ke Gecko" GET /assets/css/print.css "-" "" "HTTP/1.1" xconnx7660 "-" "-""

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.