# Not parsing apache access and error logs **URL:** \ **Category:** Logstash\ **Created:** [January 28, 2021, 4:45pm UTC](https://discuss.elastic.co/t/not-parsing-apache-access-and-error-logs/262535 "2021-01-28T16:45:39Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![kevintts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevintts/32/86660_2.png) [@kevintts](https://discuss.elastic.co/u/kevintts)\ **Post date:** [January 28, 2021, 4:45pm UTC](https://discuss.elastic.co/t/not-parsing-apache-access-and-error-logs/262535/1 "2021-01-28T16:45:39Z")
I am using filebeat to send apache logs to logstash on my elastic server. I can see the log entries with `event.dataset` set to `apache.access` , but they aren't being parsed, so no info in dashboard apache screens. Also, when Iook at Observability...logs on the demo system it has the `[apache][access]` prefix per line and when I "view details" it shows fields like `http.request.method`. Both machines Centos 7 ``` ELK machine elasticsearch.x86_64 7.9.0-1 filebeat.x86_64 7.9.0-1 kibana.x86_64 7.9.0-1 logstash.noarch 1:7.9.0-1 metricbeat.x86_64 7.9.0-1 Apache machine filebeat.x86_64 7.9.0-1 metricbeat.x86_64 7.9.0-1 ``` In my apache host file, I have made sure the format as required. ```auto LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined CustomLog /var/log/httpd/mysystem-access.log combined ``` `/etc/filebeat/modules.d/apache.yml` on the web server: ```auto # Module: apache # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.9/filebeat-module-apache.html - module: apache # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: ["/var/log/httpd/mysystem-access.log"] # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: ["/var/log/httpd/mysystem-error.log"] ``` On ELK machine, `ll /etc/logstash/conf.d/` ```auto -rw-r--r--. 1 root root 330 Sep 1 21:06 02-beats-input.conf -rw-r--r--. 1 root root 1426 Jan 28 16:07 10-apache-filter.conf -rw-r--r--. 1 root root 3148 Jul 31 18:08 10-syslog-filter.conf -rw-r--r--. 1 root root 169 Sep 4 23:13 30-elasticsearch-output.conf ``` `/etc/logstash/conf.d/10-apache-filter.conf` ```auto filter { if [type] in ["apache" , "apache_access" , "apache-access"] { grok { match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}", "message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}" ] overwrite => ["message"] } mutate { convert => ["response", "integer"] convert => ["bytes", "integer"] convert => ["responsetime", "float"] } geoip { source => "clientip" target => "geoip" add_tag => ["apache-geoip"] } date { match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"] remove_field => ["timestamp"] } useragent { source => "agent" } } if [type] in ["apache_error","apache-error"] { grok { match => ["message", "\[%{WORD:dayname} %{WORD:month} %{DATA:day} %{DATA:hour}:%{DATA:minute}:%{DATA:second} %{YEAR:year}\] \[%{NOTSPACE:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}"] overwrite => ["message"] } mutate { add_field => { "time_stamp" => "%{day}/%{month}/%{year}:%{hour}:%{minute}:%{second}" } } date { match => ["time_stamp", "dd/MMM/YYYY:HH:mm:ss"] remove_field => ["time_stamp","day","dayname","month","hour","minute","second","year"] } } } ``` When I go to the Grok debugger and have Sample data taken from the log shown on Observability...Logs ```auto 127.0.0.1 - - [28/Jan/2021:16:05:49 +0000] "GET /server-status?auto= HTTP/1.1" 200 482 "-" "Go-http-client/1.1 ``` Grok pattern ```auto %{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields} ``` I get the correctly parsed ```auto { "request": "/server-status?auto=", "agent": "\"Go-http-client/1.1\"", "auth": "-", "ident": "-", "verb": "GET", "referrer": "\"-\"", "response": "200", "bytes": "482", "clientip": "127.0.0.1", "httpversion": "1.1", "extra_fields": "", "timestamp": "28/Jan/2021:16:05:49 +0000" } ``` So it would appear that the filter isn't firing. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 28, 2021, 5:08pm UTC](https://discuss.elastic.co/t/not-parsing-apache-access-and-error-logs/262535/2 "2021-01-28T17:08:07Z")
Apologies.. Missread the question. Perhaps I am a little confused... if you are using one of the supported formats you should not need to parse, or are you intentionally parsing this your self. Question for a test did you run `filebeat setup -e` [this](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html#setup-assets) is crucial to make sure the correct ingest pipelines were loaded. I often test sending from Filebeat directly to Elasticsearch first before passing trough logstash as a test. Did you try to just sending directly from filebeat to Elasticsearch as a test and see if you are getting correct parsing? If so then you don't need to do any parsing in logstash you just use it as a pass through.. It could look like this... this treats logstash as a pass through and then will use the automatically configured ingest pipelines to parse the apache logs... if they are one of the supported formats. ``` ################################################ # beats->logstash->es default config. ################################################ input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" pipeline => "%{[@metadata][pipeline]}" user => "elastic" password => "secret" } } else { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" user => "elastic" password => "secret" } } } ``` Also as a note when you set up your conf.d that way think of all the .conf files being concatenated... which I think you get. ---
**Author:** ![kevintts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevintts/32/86660_2.png) [@kevintts](https://discuss.elastic.co/u/kevintts)\ **Post date:** [January 29, 2021, 2:58pm UTC](https://discuss.elastic.co/t/not-parsing-apache-access-and-error-logs/262535/3 "2021-01-29T14:58:49Z")
Thanks for your reply. I was definitely confused. This started all started off at version 7.8 some time ago, Then I accidentally upgraded some items to 7.9 so I updated everything to 7.9, so possibly things are a bit half -baked. In the end, I did the `filebeat setup -e` on the ELK machine as I couldn't get the permissions sorted to do it from my apache machine. this added the apache ingest pipeline which was previously missing. I then re-did my `logstash` `modules.d` directory to reflect the outputs of your example and removed my apache filter from there. It now works ok. Many thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2021, 2:59pm UTC](https://discuss.elastic.co/t/not-parsing-apache-access-and-error-logs/262535/4 "2021-02-26T14:59:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.