Unable to ingest apache2 logs through filebeat


(Nishanth Raj) #1

Hi Team,
I am trying to ingest the apache 2 access and error logs through the filebeat index and getting the below error

./filebeat -e -c filebeat.yml -d "publish"
2018-08-28T02:40:42.896-0700    INFO    instance/beat.go:468    Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-08-28T02:40:42.896-0700    INFO    instance/beat.go:475    Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5
2018-08-28T02:40:42.896-0700    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.4
2018-08-28T02:40:42.896-0700    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-08-28T02:40:42.896-0700    INFO    pipeline/module.go:76   Beat name: STOVLNX3124.corp.frk.com
2018-08-28T02:40:42.899-0700    INFO    beater/filebeat.go:62   Enabled modules/filesets: apache2 (access, error),  ()
2018-08-28T02:40:42.899-0700    ERROR   instance/beat.go:667    Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log
Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log

and my filebeat.yml file look like below

#==========================  Modules configuration ============================
filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: /opt/bea/Logwarhouse/MarketApache/*access.log


  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: /opt/bea/Logwarhouse/MarketApache/*error.log


#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/modules.d/apache2.yml

  # Set to true to enable config reloading
  #reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

  #setup.template.settings:
  #index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "elastic"

(Nishanth Raj) #2

In addition to the above setup, i executed the below command as well and also installed ingest-user-agent and ingest-geoip..

 ./filebeat -e -M "apache2.access.var.paths=[/opt/bea/Logwarhouse/MarketApache/*access.log]" -M "apache2.error.var.paths=[/opt/bea/Logwarhouse/MarketApache/*error.log]"
2018-08-28T05:50:50.573-0700    INFO    instance/beat.go:468    Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-08-28T05:50:50.574-0700    INFO    instance/beat.go:475    Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5
2018-08-28T05:50:50.574-0700    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.4
2018-08-28T05:50:50.574-0700    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-08-28T05:50:50.575-0700    INFO    pipeline/module.go:76   Beat name: STOVLNX3124.corp.frk.com
2018-08-28T05:50:50.578-0700    INFO    beater/filebeat.go:62   Enabled modules/filesets: apache2 (access, error),  ()
2018-08-28T05:50:50.579-0700    INFO    instance/beat.go:301    filebeat start running.
2018-08-28T05:50:50.579-0700    INFO    registrar/registrar.go:110      Loading registrar data from /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data/registry
2018-08-28T05:50:50.579-0700    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-08-28T05:50:50.579-0700    INFO    registrar/registrar.go:121      States Loaded from registrar: 3
2018-08-28T05:50:50.579-0700    INFO    crawler/crawler.go:48   Loading Prospectors: 2
2018-08-28T05:50:50.580-0700    INFO    log/prospector.go:111   Configured paths: [/opt/bea/Logwarhouse/MarketApache/*access.log]
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/uk-access.log
2018-08-28T05:50:50.581-0700    INFO    log/prospector.go:111   Configured paths: [/opt/bea/Logwarhouse/MarketApache/*error.log]
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/us-dev-3-access.log
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-qa-access.log
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/ind-dev-1-access.log
2018-08-28T05:50:50.581-0700    INFO    log/harvester.go:216    Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-dev4-access.log
2018-08-28T05:50:50.583-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/error.log*]
2018-08-28T05:50:50.584-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*]
2018-08-28T05:50:50.584-0700    INFO    crawler/crawler.go:82   Loading and starting Prospectors completed. Enabled prospectors: 2
2018-08-28T05:50:50.584-0700    INFO    cfgfile/reload.go:127   Config reloader started
2018-08-28T05:50:50.587-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*]
2018-08-28T05:50:50.588-0700    INFO    log/prospector.go:111   Configured paths: [/var/log/apache2/error.log*]
2018-08-28T05:50:50.588-0700    INFO    cfgfile/reload.go:258   Starting 1 runners ...
2018-08-28T05:50:50.588-0700    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-08-28T05:50:50.591-0700    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.4
2018-08-28T05:50:50.592-0700    ERROR   fileset/factory.go:93   Error loading pipeline: Error loading pipeline for fileset apache2/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
    sudo bin/elasticsearch-plugin install ingest-user-agent
    sudo bin/elasticsearch-plugin install ingest-geoip

bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%  
-> Installed ingest-user-agent


 bin/elasticsearch-plugin install ingest-geoip
-> Downloading ingest-geoip from elastic
[=================================================] 100%  
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip

(Steffen Siering) #3

The vars.paths setting in the apache module requires an array. The template engine complaints about it being a plain string.

Update your modules setting to say:

filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - '/opt/bea/Logwarhouse/MarketApache/*access.log'


  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - '/opt/bea/Logwarhouse/MarketApache/*error.log'


(Nishanth Raj) #4

Thank you Steffen, I am able to create filebeat index for apache access logs now, but although i have included the error log in above file, it was not picked up. May i know what would be the possible cause for this.


(Steffen Siering) #5

May i know what would be the possible cause for this.

You mean the cause for th error message Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log ?

Beats uses a template engine to expand user settings into a configuration object. The template requires var.paths to be a list of strings. You originally did not configure a list of string, but a string value only. Normally the list of strings is iterated, creating one configuration per entry. A single string can not be iterated. It's basically a type error.


(Nishanth Raj) #6

No Not for this, I am saying apache error logs are getting picked up by the index.


(Steffen Siering) #7

Can you share filebeat logs? Have you checked that the path is correct?


(Nishanth Raj) #8

Hi Steffens,
Let me park the above discussion. I have some additonal details captured in the apache access log and i would like to know if the filebeat can parse them as well.

Myapache access looks like below.

0 10.65.19.201 - - [22/May/2018:00:08:50 -0700] "GET /assets/css/print.css HTTP/1.1" 200 509 "https://roddoc.patqa.corp.rrk.com/profile-web/signin" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) li
ke Gecko" GET /assets/css/print.css "-" "" "HTTP/1.1" xconnx7660 "-" "-""


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.