# Apache grok filter not working the way I expect it to. Should I be using a different approach? **URL:** \ **Category:** Logstash\ **Created:** [March 19, 2018, 6:08pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625 "2018-03-19T18:08:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![William-Robert](https://avatars.discourse-cdn.com/v4/letter/w/ccd318/32.png) [@William-Robert](https://discuss.elastic.co/u/William-Robert)\ **Post date:** [March 19, 2018, 6:08pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625/1 "2018-03-19T18:08:04Z")
I'm trying to set up filebeat/ELK such that I can drop an apache log file into a directory and it will be indexed automatically. I'm currently succeeding in having the files indexed, however, each log file is just being indexed at one long string. For example, this is copied from my kibana dashboard: |Time |message |source | |March 19th 2018, 12:53:35.172| 66.249.79.132 - - [26/Feb/2018:10:34:58 +0000] "GET /places/businesses/amc/ HTTP/1.0" 404 60 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"| /home/me/apache.log I'm expecting these logs to be indexed such that each component is an individual field instead of just one long string. Maybe I'm misunderstanding how to use the grok filter, or there are additional settings I need to include. For example, perhaps I did not properly install the geoip plugin??? I've tried using "%{COMMONAPACHELOG}" and "%{COMBINEDPACHELOG}" but I still get the same results, the log files are indexed as just one string. Here are my logstash conf files: 02-beats-input.conf ``` input { beats { port => 5044 } } ``` 10-syslog-filter.conf ``` filter { if [type] in ["apache" , "apache_access" , "apache-access", "syslog", "log"] { grok { match => { "message" => "%{COMMONAPACHELOG}" } } mutate { convert => ["response", "integer"] convert => ["bytes", "integer"] convert => ["responsetime", "float"] } geoip { source => "clientip" target => "geoip" add_tag => ["apache-geoip"] } date { match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"] remove_field => ["timestamp"] } useragent { source => "agent" } } if [type] in ["apache_error","apache-error"] { grok { match => ["message", "\[%{WORD:dayname} %{WORD:month} %{DATA:day} %{DATA:hour}:% {DATA:minute}:%{DATA:second} %{YEAR:year}\] \[%{NOTSPACE:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}"] overwrite => ["message"] } mutate { add_field => { "time_stamp" => "%{day}/%{month}/%{year}:%{hour}:%{minute}:%{second}" } } date { match => ["time_stamp", "dd/MMM/YYYY:HH:mm:ss"] remove_field => ["time_stamp","day","dayname","month","hour","minute","second","year"] } } } ``` 30-elasticsearch-output.conf ``` output { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 19, 2018, 6:31pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625/2 "2018-03-19T18:31:20Z")
Please show a complete example event. Copy/paste from Kibana's JSON tab. ---
**Author:** ![William-Robert](https://avatars.discourse-cdn.com/v4/letter/w/ccd318/32.png) [@William-Robert](https://discuss.elastic.co/u/William-Robert)\ **Post date:** [March 19, 2018, 6:38pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625/3 "2018-03-19T18:38:22Z")
Is this what you're asking for? ``` { "_index": "filebeat-2018.03.19", "_type": "doc", "_id": "AWI_hI6QvVPbTbcX3xuE", "_version": 1, "_score": null, "_source": { "@timestamp": "2018-03-19T18:27:49.176Z", "beat": { "hostname": "mycomp", "name": "mycomp", "version": "5.6.8" }, "input_type": "log", "message": "174.141.131.19 - - [22/Feb/2018:00:14:02 +0000] \"POST /wp-admin/admin-ajax.php HTTP/1.0\" 200 123 \"https://SOMEDOMAIN.com/wp-admin/post.php?post=11080&action=edit\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36\"", "offset": 5825514, "source": "/home/me/ftflogs/apache9.log", "type": "log" }, "fields": { "@timestamp": [ 1521484069176 ] }, "sort": [ 1521484069176 ] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 19, 2018, 7:01pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625/4 "2018-03-19T19:01:09Z")
Things look correct from here. Comment out the conditionals to see if those are what's stopping the filtering. (Unrelated to your problem, but what you have is a combined log, not a common log.) ---
**Author:** ![William-Robert](https://avatars.discourse-cdn.com/v4/letter/w/ccd318/32.png) [@William-Robert](https://discuss.elastic.co/u/William-Robert)\ **Post date:** [March 26, 2018, 4:31pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625/5 "2018-03-26T16:31:08Z")
In case anyone runs into this issue themselves, I figured out the problem. Within my filebeat.yml file, I was sending data directly to elasticsearch and NOT logstash. I commented out the elasticsearch output options, and uncommented logstash, set the path and now it's working correctly. Thanks for helping me work through the problem!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 23, 2018, 4:31pm UTC](https://discuss.elastic.co/t/apache-grok-filter-not-working-the-way-i-expect-it-to-should-i-be-using-a-different-approach/124625/6 "2018-04-23T16:31:37Z")
Closed on April 23, 2018, 4:31pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.