Hi Team,
I am trying to ingest the apache 2 access and error logs through the filebeat index and getting the below error
./filebeat -e -c filebeat.yml -d "publish"
2018-08-28T02:40:42.896-0700 INFO instance/beat.go:468 Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-08-28T02:40:42.896-0700 INFO instance/beat.go:475 Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5
2018-08-28T02:40:42.896-0700 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2018-08-28T02:40:42.896-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200
2018-08-28T02:40:42.896-0700 INFO pipeline/module.go:76 Beat name: STOVLNX3124.corp.frk.com
2018-08-28T02:40:42.899-0700 INFO beater/filebeat.go:62 Enabled modules/filesets: apache2 (access, error), ()
2018-08-28T02:40:42.899-0700 ERROR instance/beat.go:667 Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log
Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log
and my filebeat.yml file look like below
#========================== Modules configuration ============================
filebeat.modules:
#------------------------------- Apache2 Module ------------------------------
- module: apache2
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: /opt/bea/Logwarhouse/MarketApache/*access.log
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: /opt/bea/Logwarhouse/MarketApache/*error.log
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/modules.d/apache2.yml
# Set to true to enable config reloading
#reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
#setup.template.settings:
#index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "elastic"
In addition to the above setup, i executed the below command as well and also installed ingest-user-agent and ingest-geoip..
./filebeat -e -M "apache2.access.var.paths=[/opt/bea/Logwarhouse/MarketApache/*access.log]" -M "apache2.error.var.paths=[/opt/bea/Logwarhouse/MarketApache/*error.log]"
2018-08-28T05:50:50.573-0700 INFO instance/beat.go:468 Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-08-28T05:50:50.574-0700 INFO instance/beat.go:475 Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5
2018-08-28T05:50:50.574-0700 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2018-08-28T05:50:50.574-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200
2018-08-28T05:50:50.575-0700 INFO pipeline/module.go:76 Beat name: STOVLNX3124.corp.frk.com
2018-08-28T05:50:50.578-0700 INFO beater/filebeat.go:62 Enabled modules/filesets: apache2 (access, error), ()
2018-08-28T05:50:50.579-0700 INFO instance/beat.go:301 filebeat start running.
2018-08-28T05:50:50.579-0700 INFO registrar/registrar.go:110 Loading registrar data from /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data/registry
2018-08-28T05:50:50.579-0700 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-08-28T05:50:50.579-0700 INFO registrar/registrar.go:121 States Loaded from registrar: 3
2018-08-28T05:50:50.579-0700 INFO crawler/crawler.go:48 Loading Prospectors: 2
2018-08-28T05:50:50.580-0700 INFO log/prospector.go:111 Configured paths: [/opt/bea/Logwarhouse/MarketApache/*access.log]
2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/uk-access.log
2018-08-28T05:50:50.581-0700 INFO log/prospector.go:111 Configured paths: [/opt/bea/Logwarhouse/MarketApache/*error.log]
2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/us-dev-3-access.log
2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-qa-access.log
2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/ind-dev-1-access.log
2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-dev4-access.log
2018-08-28T05:50:50.583-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/error.log*]
2018-08-28T05:50:50.584-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*]
2018-08-28T05:50:50.584-0700 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 2
2018-08-28T05:50:50.584-0700 INFO cfgfile/reload.go:127 Config reloader started
2018-08-28T05:50:50.587-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*]
2018-08-28T05:50:50.588-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/error.log*]
2018-08-28T05:50:50.588-0700 INFO cfgfile/reload.go:258 Starting 1 runners ...
2018-08-28T05:50:50.588-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200
2018-08-28T05:50:50.591-0700 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
2018-08-28T05:50:50.592-0700 ERROR fileset/factory.go:93 Error loading pipeline: Error loading pipeline for fileset apache2/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
sudo bin/elasticsearch-plugin install ingest-user-agent
sudo bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%
-> Installed ingest-user-agent
bin/elasticsearch-plugin install ingest-geoip
-> Downloading ingest-geoip from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
Continue with installation? [y/N]y
-> Installed ingest-geoip
The vars.paths setting in the apache module requires an array. The template engine complaints about it being a plain string.
Update your modules setting to say:
filebeat.modules:
#------------------------------- Apache2 Module ------------------------------
- module: apache2
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths:
- '/opt/bea/Logwarhouse/MarketApache/*access.log'
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths:
- '/opt/bea/Logwarhouse/MarketApache/*error.log'
Thank you Steffen, I am able to create filebeat index for apache access logs now, but although i have included the error log in above file, it was not picked up. May i know what would be the possible cause for this.
May i know what would be the possible cause for this.
You mean the cause for th error message Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log ?
Beats uses a template engine to expand user settings into a configuration object. The template requires var.paths to be a list of strings. You originally did not configure a list of string, but a string value only. Normally the list of strings is iterated, creating one configuration per entry. A single string can not be iterated. It's basically a type error.
Hi Steffens,
Let me park the above discussion. I have some additonal details captured in the apache access log and i would like to know if the filebeat can parse them as well.
Myapache access looks like below.
0 10.65.19.201 - - [22/May/2018:00:08:50 -0700] "GET /assets/css/print.css HTTP/1.1" 200 509 "https://roddoc.patqa.corp.rrk.com/profile-web/signin" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) li
ke Gecko" GET /assets/css/print.css "-" "" "HTTP/1.1" xconnx7660 "-" "-""
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.