# Lotstash is not picking Apache Logs config **URL:** \ **Category:** Logstash\ **Created:** [October 9, 2019, 1:01am UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747 "2019-10-09T01:01:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![sud0](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@sud0](https://discuss.elastic.co/u/sud0)\ **Post date:** [October 9, 2019, 1:01am UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/1 "2019-10-09T01:01:39Z")
I have two Apache webservers (1x uat and 1x prod). I am sending the logs to my ELK setup for both of them. For the UAT one, it is working fine. For the PROD one, the logs are now shown in Kibana. I need some help to debug it, and find out where the issue is. I'll post some config files below so you guys have an idea how this is setup. - The `LogFormat` config on the `httpd.conf` is the same for both environments. - PROD Apache version: Apache/2.2.15 (Unix) - UAT Apache version: Apache/2.2.9 (Unix) /etc/logstash/conf.d/logstash.conf: ``` input { ## PROD file { type => "apache_access_log" start_position => "beginning" path => "/mnt/logs/web/access_log" } ## UAT file { type => "uat_apache_access_log" start_position => "beginning" path => "/mnt/logs/uatweb/access_log" } } filter { # Remove unwanted carrage returns, global to all filter types mutate { gsub => ['message', "\r", ''] } ###################################################### # PROD # Apache access filter if [type] == "apache_access_log" { mutate { replace => { 'host' => 'webserver.datacentre.example.com' } add_field => { 'environment' => 'production' 'service' => 'apache_access' } } grok { match => { "message" => "%{IPORHOST:clientip}%{SPACE}\[%{HTTPDATE:timestamp}\]%{SPACE}%{NUMBER:port}%{SPACE}%{WORD:method}%{SPACE}%{URIPATHPARAM:request_uri}%{SPACE}%{NOTSPACE}%{SPACE}%{NUMBER:status_code}%{SPACE}%{NOTSPACE:bytes_delivered}%{SPACE}%{NUMBER:duration%}%{SPACE}(?:%{URI:referrer}|.*)%{SPACE}%{QS:agent}%{SPACE}%{GREEDYDATA:general_data}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target => "@timestamp" } } # UAT # Apache access filter if [type] == "uat_apache_access_log" { mutate { replace => { 'host' => 'uatweb.datacentre.example.com' } add_field => { 'environment' => 'uat' 'service' => 'apache_access' } } grok { match => { "message" => "%{IPORHOST:clientip}%{SPACE}\[%{HTTPDATE:timestamp}\]%{SPACE}%{NUMBER:port}%{SPACE}%{WORD:method}%{SPACE}%{URIPATHPARAM:request_uri}%{SPACE}%{NOTSPACE}%{SPACE}%{NUMBER:status_code}%{SPACE}%{NOTSPACE:bytes_delivered}%{SPACE}%{NUMBER:duration%}%{SPACE}(?:%{URI:referrer}|.*)%{SPACE}%{QS:agent}%{SPACE}%{GREEDYDATA:general_data}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target => "@timestamp" } } output { elasticsearch { hosts => ["localhost:9200"] # Weekly index (for pruning) index => "mw-log-index-%{+YYYY.'w'ww}" } stdout { codec => rubydebug } } ``` On KIbana, there is NO logs for PROD: ![Screenshot%20from%202019-10-09%2013-53-04](https://us1.discourse-cdn.com/elastic/original/3X/2/3/2317e31a2a228c2baee2dbee69a5de228bf0c2f9.png) ... however for UAT there are: ![Screenshot%20from%202019-10-09%2013-53-41](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a7d5ce0aae06e8d9c8229059f1535014c9dd82a.png) /var/log/logstash/logstash-plain.log: ```` [2019-10-09T13:45:04,253][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled filter P[filter-mutate{"replace"=>{"host"=>"webserver.datacentre.example.com"}, "add_field"=>{"environment"=>"production", "service"=>"apache_access"}}|[str]pipeline:209:9:``` mutate { replace => { 'host' => 'webserver.datacentre.example.com' } add_field => { 'environment' => 'production' 'service' => 'apache_access' } } ```] ```` How can I troubleshoot this? where to start looking? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 9, 2019, 2:22pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/2 "2019-10-09T14:22:21Z")
In the UAT section you are testing ``` if [type] == "apache_access_log" ``` which looks wrong to me. Should that be looking for "uat\_apache\_access\_log"? ---
**Author:** ![sud0](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@sud0](https://discuss.elastic.co/u/sud0)\ **Post date:** [October 9, 2019, 8:32pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/3 "2019-10-09T20:32:32Z")
Sorry my bad.... I copied it wrong. I've fixed the post. (it is already like `uat_apache_access_log`. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 9, 2019, 8:38pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/4 "2019-10-09T20:38:42Z")
OK, try filtering for NOT environment: uat and see if you can find the missing data. Also, try looking across a much longer period of time. ---
**Author:** ![sud0](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@sud0](https://discuss.elastic.co/u/sud0)\ **Post date:** [October 9, 2019, 8:41pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/5 "2019-10-09T20:41:56Z")
Ok Thanks for your suggestion... but nothing new! ![Screenshot%20from%202019-10-10%2009-41-20](https://us1.discourse-cdn.com/elastic/original/3X/c/3/c377a914a11dfb6425fe2cc84b9452d3cb8501dc.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 9, 2019, 8:49pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/6 "2019-10-09T20:49:55Z")
It sounds as though the file input is not reading the file. Enable tracing as described in [this](https://discuss.elastic.co/t/input-simple-txt-file/202892/4) post and see what filewatch has to say. ---
**Author:** ![sud0](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@sud0](https://discuss.elastic.co/u/sud0)\ **Post date:** [October 9, 2019, 9:05pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/7 "2019-10-09T21:05:30Z")
Done! Let me know if you need more log: ```` [2019-10-10T09:54:27,331][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@start_position = "beginning" [2019-10-10T09:54:27,331][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@path = ["/mnt/logs/web/access_log"] [2019-10-10T09:54:27,331][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@id = "c3b85ae40ef876422eb8f30486cf9828a2903d75d01bb103edbe6da301cc4f38" [2019-10-10T09:54:27,331][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@type = "apache_access_log" [2019-10-10T09:54:27,331][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@enable_metric = true [2019-10-10T09:54:27,336][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@codec = "plain_1f0e86ee-70e6-4bc1-b9ca-ad67c2c7c570", enable_metric=>true, charset=>"UTF-8"> [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@add_field = {} [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@stat_interval = 1.0 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@discover_interval = 15 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@sincedb_write_interval = 15.0 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@delimiter = "\n" [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@close_older = 3600.0 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@mode = "tail" [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@file_completed_action = "delete" [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@sincedb_clean_after = 1209600.0 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@file_chunk_size = 32768 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@file_chunk_count = 140737488355327 [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@file_sort_by = "last_modified" [2019-10-10T09:54:27,337][DEBUG][logstash.inputs.file] config LogStash::Inputs::File/@file_sort_direction = "asc" [2019-10-10T09:54:27,353][DEBUG][logstash.codecs.plain] config LogStash::Codecs::Plain/@id = "plain_7e9cf891-01b9-400c-8e93-087db79f785e" [2019-10-10T09:54:27,354][DEBUG][logstash.codecs.plain] config LogStash::Codecs::Plain/@enable_metric = true [2019-10-10T09:54:27,354][DEBUG][logstash.codecs.plain] config LogStash::Codecs::Plain/@charset = "UTF-8" [2019-10-10T09:54:35,199][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional [if (event.getField('[type]')=='apache_access_log')] into org.logstash.config.ir.compiler.ComputeStepSyntaxElement@56d811ee [2019-10-10T09:54:35,207][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional [if (event.getField('[type]')=='apache_access_log')] into org.logstash.config.ir.compiler.ComputeStepSyntaxElement@56d811ee [2019-10-10T09:54:35,217][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional [if (event.getField('[type]')=='apache_access_log')] into org.logstash.config.ir.compiler.ComputeStepSyntaxElement@56d811ee [2019-10-10T09:54:35,232][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional [if (event.getField('[type]')=='apache_access_log')] into org.logstash.config.ir.compiler.ComputeStepSyntaxElement@56d811ee [2019-10-10T09:54:35,249][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional [if (event.getField('[type]')=='apache_access_log')] into org.logstash.config.ir.compiler.ComputeStepSyntaxElement@56d811ee [2019-10-10T09:54:35,278][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled filter P[filter-mutate{"replace"=>{"host"=>"webserver.datacentre.example.com"}, "add_field"=>{"environment"=>"production", "service"=>"apache_access"}}|[str]pipeline:209:9:``` mutate { replace => { 'host' => 'webserver.datacentre.example.com' } add_field => { 'environment' => 'production' 'service' => 'apache_access' } } ```] into org.logstash.config.ir.compiler.ComputeStepSyntaxElement@6ccb8168 ```` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 9, 2019, 9:45pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/8 "2019-10-09T21:45:04Z")
We need the TRACE messages from filewatch, which look like this ``` [2019-07-30T13:18:09,252][TRACE][filewatch.tailmode.processor] Delayed Delete processing [2019-07-30T13:18:09,267][TRACE][filewatch.tailmode.processor] Watched + Active restat processing [2019-07-30T13:18:09,297][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-07-30T13:18:09,358][TRACE][filewatch.tailmode.processor] Rotation In Progress processing ``` There should be a lot of them. ---
**Author:** ![sud0](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@sud0](https://discuss.elastic.co/u/sud0)\ **Post date:** [October 9, 2019, 10:16pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/9 "2019-10-09T22:16:22Z")
> [@Badger](#): > > filewatch You can grab the log file [here](https://we.tl/t-HhIdSUfXU9). Cheers! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 9, 2019, 11:08pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/10 "2019-10-09T23:08:39Z")
The string /web/access\_log never occurs in that file, which tells me that logstash never sees the file. Are you sure that the name is right and that logstash has execute access to that directory? ---
**Author:** ![sud0](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@sud0](https://discuss.elastic.co/u/sud0)\ **Post date:** [October 9, 2019, 11:21pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/11 "2019-10-09T23:21:17Z")
You're right. The path `/mnt/logs/web/` was mounted as `nobody:nobody`, therefore, the user `logstash` did not have permission. Adding the `Domain = localdomain` config in the `/etc/idmapd.conf` file and re-mounting the NFS volume fixed my problem. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 6, 2019, 11:21pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747/12 "2019-11-06T23:21:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.