# Unable to ingest apache2 logs through filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 28, 2018, 9:46am UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309 "2018-08-28T09:46:19Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Post date:** [August 28, 2018, 9:46am UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/1 "2018-08-28T09:46:19Z")
Hi Team, I am trying to ingest the apache 2 access and error logs through the filebeat index and getting the below error ``` ./filebeat -e -c filebeat.yml -d "publish" 2018-08-28T02:40:42.896-0700 INFO instance/beat.go:468 Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs] 2018-08-28T02:40:42.896-0700 INFO instance/beat.go:475 Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5 2018-08-28T02:40:42.896-0700 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4 2018-08-28T02:40:42.896-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200 2018-08-28T02:40:42.896-0700 INFO pipeline/module.go:76 Beat name: STOVLNX3124.corp.frk.com 2018-08-28T02:40:42.899-0700 INFO beater/filebeat.go:62 Enabled modules/filesets: apache2 (access, error), () 2018-08-28T02:40:42.899-0700 ERROR instance/beat.go:667 Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log ``` and my filebeat.yml file look like below ``` #========================== Modules configuration ============================ filebeat.modules: #------------------------------- Apache2 Module ------------------------------ - module: apache2 access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: /opt/bea/Logwarhouse/MarketApache/*access.log error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: /opt/bea/Logwarhouse/MarketApache/*error.log #============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/modules.d/apache2.yml # Set to true to enable config reloading #reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== #setup.template.settings: #index.number_of_shards: 3 #index.codec: best_compression #_source.enabled: false #================================ Outputs ===================================== #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" username: "elastic" password: "elastic" ``` ---
**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Post date:** [August 28, 2018, 12:56pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/2 "2018-08-28T12:56:17Z")
In addition to the above setup, i executed the below command as well and also installed ingest-user-agent and ingest-geoip.. ``` ./filebeat -e -M "apache2.access.var.paths=[/opt/bea/Logwarhouse/MarketApache/*access.log]" -M "apache2.error.var.paths=[/opt/bea/Logwarhouse/MarketApache/*error.log]" 2018-08-28T05:50:50.573-0700 INFO instance/beat.go:468 Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs] 2018-08-28T05:50:50.574-0700 INFO instance/beat.go:475 Beat UUID: 0a5344b3-350e-4e02-aeee-922b98c997c5 2018-08-28T05:50:50.574-0700 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4 2018-08-28T05:50:50.574-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200 2018-08-28T05:50:50.575-0700 INFO pipeline/module.go:76 Beat name: STOVLNX3124.corp.frk.com 2018-08-28T05:50:50.578-0700 INFO beater/filebeat.go:62 Enabled modules/filesets: apache2 (access, error), () 2018-08-28T05:50:50.579-0700 INFO instance/beat.go:301 filebeat start running. 2018-08-28T05:50:50.579-0700 INFO registrar/registrar.go:110 Loading registrar data from /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data/registry 2018-08-28T05:50:50.579-0700 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s 2018-08-28T05:50:50.579-0700 INFO registrar/registrar.go:121 States Loaded from registrar: 3 2018-08-28T05:50:50.579-0700 INFO crawler/crawler.go:48 Loading Prospectors: 2 2018-08-28T05:50:50.580-0700 INFO log/prospector.go:111 Configured paths: [/opt/bea/Logwarhouse/MarketApache/*access.log] 2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/uk-access.log 2018-08-28T05:50:50.581-0700 INFO log/prospector.go:111 Configured paths: [/opt/bea/Logwarhouse/MarketApache/*error.log] 2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/us-dev-3-access.log 2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-qa-access.log 2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/ind-dev-1-access.log 2018-08-28T05:50:50.581-0700 INFO log/harvester.go:216 Harvester started for file: /opt/bea/Logwarhouse/MarketApache/iws-dev4-access.log 2018-08-28T05:50:50.583-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/error.log*] 2018-08-28T05:50:50.584-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*] 2018-08-28T05:50:50.584-0700 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 2 2018-08-28T05:50:50.584-0700 INFO cfgfile/reload.go:127 Config reloader started 2018-08-28T05:50:50.587-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log*] 2018-08-28T05:50:50.588-0700 INFO log/prospector.go:111 Configured paths: [/var/log/apache2/error.log*] 2018-08-28T05:50:50.588-0700 INFO cfgfile/reload.go:258 Starting 1 runners ... 2018-08-28T05:50:50.588-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200 2018-08-28T05:50:50.591-0700 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4 2018-08-28T05:50:50.592-0700 ERROR fileset/factory.go:93 Error loading pipeline: Error loading pipeline for fileset apache2/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes: sudo bin/elasticsearch-plugin install ingest-user-agent sudo bin/elasticsearch-plugin install ingest-geoip bin/elasticsearch-plugin install ingest-user-agent -> Downloading ingest-user-agent from elastic [=================================================] 100% -> Installed ingest-user-agent bin/elasticsearch-plugin install ingest-geoip -> Downloading ingest-geoip from elastic [=================================================] 100% @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.reflect.ReflectPermission suppressAccessChecks See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y -> Installed ingest-geoip ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 28, 2018, 2:40pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/3 "2018-08-28T14:40:46Z")
The `vars.paths` setting in the apache module requires an array. The template engine complaints about it being a plain string. Update your modules setting to say: ```auto filebeat.modules: #------------------------------- Apache2 Module ------------------------------ - module: apache2 access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: - '/opt/bea/Logwarhouse/MarketApache/*access.log' error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: - '/opt/bea/Logwarhouse/MarketApache/*error.log' ``` ---
**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Post date:** [August 29, 2018, 1:36pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/4 "2018-08-29T13:36:41Z")
> [@Nithani25](#): > > ./filebeat -e -M "apache2.access.var.paths=[/opt/bea/Logwarhouse/MarketApache/\*access.log]" -M "apache2.error.var.paths=[/opt/bea/Logwarhouse/MarketApache/\*error.log]" Thank you Steffen, I am able to create filebeat index for apache access logs now, but although i have included the error log in above file, it was not picked up. May i know what would be the possible cause for this. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 29, 2018, 5:41pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/5 "2018-08-29T17:41:46Z")
> May i know what would be the possible cause for this. You mean the cause for th error message `Exiting: Error getting config for fielset apache2/access: Error interpreting the template of the prospector: template: text:3:22: executing "text" at <.paths>: range can't iterate over /opt/bea/Logwarhouse/MarketApache/*access.log` ? Beats uses a template engine to expand user settings into a configuration object. The template requires `var.paths` to be a list of strings. You originally did not configure a list of string, but a string value only. Normally the list of strings is iterated, creating one configuration per entry. A single string can not be iterated. It's basically a type error. ---
**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Post date:** [August 30, 2018, 5:09am UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/6 "2018-08-30T05:09:12Z")
No Not for this, I am saying apache error logs are getting picked up by the index. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 30, 2018, 12:36pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/7 "2018-08-30T12:36:00Z")
Can you share filebeat logs? Have you checked that the path is correct? ---
**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Post date:** [September 14, 2018, 2:25pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/8 "2018-09-14T14:25:14Z")
Hi Steffens, Let me park the above discussion. I have some additonal details captured in the apache access log and i would like to know if the filebeat can parse them as well. Myapache access looks like below. 0 10.65.19.201 - - [22/May/2018:00:08:50 -0700] "GET /assets/css/print.css HTTP/1.1" 200 509 "[https://roddoc.patqa.corp.rrk.com/profile-web/signin](https://roddoc.patqa.corp.rrk.com/profile-web/signin)" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) li ke Gecko" GET /assets/css/print.css "-" "" "HTTP/1.1" xconnx7660 "-" "-"" ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 12, 2018, 2:25pm UTC](https://discuss.elastic.co/t/unable-to-ingest-apache2-logs-through-filebeat/146309/9 "2018-10-12T14:25:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.