# Logstash put all apache log entries into one index with the current date in kibana **URL:** \ **Category:** Logstash\ **Created:** [June 21, 2019, 11:23pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939 "2019-06-21T23:23:09Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [June 21, 2019, 11:23pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/1 "2019-06-21T23:23:09Z")
Using the latest versions of ELK. Input data: 100.43.83.137 - - [29/May/2014:02:12:13 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 100.43.83.137 - - [29/May/2014:02:12:17 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 100.43.83.137 - - [29/May/2014:02:12:18 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 181.141.250.44 - - [29/May/2014:02:12:26 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "[http://www.google.com/search?q=www.google.com&client=ms-android-alcatel&hl=es&source=android-launcher-widget&v=133247963&source=lnms&tbm=isch&sa=X&ei=F1sBU7ewM8\_HkAen7IH4Aw&ved=0CAgQ\_AUoAA](http://www.google.com/search?q=www.google.com&client=ms-android-alcatel&hl=es&source=android-launcher-widget&v=133247963&source=lnms&tbm=isch&sa=X&ei=F1sBU7ewM8_HkAen7IH4Aw&ved=0CAgQ_AUoAA)" "Mozilla/5.0 (Linux; U; Android 2.3.6; es-us; ALCATEL ONE TOUCH 4010A Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1" 187.211.57.202 - - [29/May/2014:02:13:01 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "[http://www.google.com.mx/search?q=www.google.com&client=ms-android-americamovil-mx&hl=es&source=android-launcher-search&v=141400000&source=lnms&tbm=isch&sa=X&ei=fk8BU5\_HMMrayAGT3YGoCQ&ved=0CAkQ\_AUoAA&biw=360&bih=615](http://www.google.com.mx/search?q=www.google.com&client=ms-android-americamovil-mx&hl=es&source=android-launcher-search&v=141400000&source=lnms&tbm=isch&sa=X&ei=fk8BU5_HMMrayAGT3YGoCQ&ved=0CAkQ_AUoAA&biw=360&bih=615)" "Mozilla/5.0 (Linux; U; Android 4.0.4; es-us; ALCATEL ONE TOUCH 5035A Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" logstash conf file: #input { stdin { codec =\> plain } } input { file { path =\> "/home/ec2-user/data/testlog" type =\> "logs" start\_position =\> "beginning" } } #output { stdout { codec =\> rubydebug } } output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> "elkcluster:9200" } } filter { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}" } } date { match =\> ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale =\> en remove\_field =\> "timestamp" } mutate{ convert =\> { "bytes" =\> "integer" } } geoip { source =\> "clientip" } useragent { source =\> "agent" target =\> "useragent" } } In Kibana, I see only one index with todays timestamp instead of 5 indices. been trying many combinations but am not able to figure out whats wrong. On the console, logstash output shows: (trimmed output) ``` "timezone" => "America/Mexico_City", "country_code3" => "MX", "city_name" => "León" }, "ident" => "-", "@timestamp" => 2014-05-29T07:13:01.000Z, "httpversion" => "1.1", "referrer" => "\"http://www.google.com.mx/search?q=www.google.com&client=ms-android-americamovil-mx&hl=es&source=android-launcher-search&v=141400000&source=lnms&tbm=isch&sa=X&ei=fk8BU5_HMMrayAGT3YGoCQ&ved=0CAkQ_AUoAA&biw=360&bih=615\"", "type" => "logs", "verb" => "GET", "request" => "/images/googledotcom.png", "clientip" => "187.211.57.202" ``` ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [June 27, 2019, 5:09pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/2 "2019-06-27T17:09:47Z")
Can someone please have a look at this issue. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 27, 2019, 5:43pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/3 "2019-06-27T17:43:50Z")
Why would you see 5 indices with that configuration? You are using the output plugins defaults which creates one ´logstash-\*´prefixed index per day. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [June 28, 2019, 3:17pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/4 "2019-06-28T15:17:31Z")
Hi Chris: I had to tear down EC2 setup as i had consumed the allowed free-tier resource limit. So in order to replicate the scenario, i did the same on local machine. logstash conf: its the same except input { file { path =\> "E:/ElasticSearch/logs1" type =\> "logs" start\_position =\> "beginning" } } logs1 file has 2 entries with dates 28th May 2011 and 27th Jun 2012: 83.149.9.216 - - [28/May/2011:16:13:42 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 183.60.215.50 - - [27/Jun/2012:08:00:00 -0400] "GET /scripts/netcat-webserver HTTP/1.1" 200 182 "-" "Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)" output i see when i run logstash: ``` "type" => "logs", "@timestamp" => 2011-05-28T21:13:42.000Z, "auth" => "-", "verb" => "GET", "@version" => "1", "response" => "200", "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"", "bytes" => 203023, "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"" ``` In Kibana i dont see 2 indices. instead there is one with TODAY's timestamp ( logstash-2019.06.28-000001). Its the same issue I faced while running the services on AWS. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [June 30, 2019, 12:23pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/6 "2019-06-30T12:23:33Z")
I am not sure if the configuration I used in apache.conf file is wrong. Its the standard Hello World type example for Logstash and i am pretty sure the grok pattern used is correct as well. Coz the output of logstash on the console correctly identifies the @timestamp but when the data is indexed into ES and read in Kibana, something is happening with the process which is not creating the output as expected. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [July 2, 2019, 6:13am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/7 "2019-07-02T06:13:11Z")
has something changed with the grok pattern I used? I know this isnt a prod issue but we are working on a POC to implement ELK stack. As pasrt of my research i found this simple program not working. Any help would be highly appreciated. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [July 3, 2019, 5:14am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/8 "2019-07-03T05:14:34Z")
I believe this works well in Elastic version 6. Is this some issue with version 7? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 3, 2019, 1:43pm UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/9 "2019-07-03T13:43:02Z")
> [@rohithsolomon](#): > > ``` > "@timestamp" => 2011-05-28T21:13:42.000Z, > > ``` If your events have @timestamp in 2011 then clearly the grok and date filters are working. You seem to be asserting that an elasticsearch output is putting that into an index named with today's timestamp. If that is the case then the configuration of the output is what we need to see. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [July 4, 2019, 7:16am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/10 "2019-07-04T07:16:56Z")
Here is the output configuration. I haven't used any fancy operations, just printed it on the console and then indexed into ES. Did i miss anything? output { stdout { codec =\> rubydebug } elasticsearch { } } ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [July 4, 2019, 8:58am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/11 "2019-07-04T08:58:04Z")
There is another issue I just noticed: input file has 2 entries as shown at the beginning of the post. When i run logstash, I see just one index on console (related to the IP 83.149.9.216). A second index (related to 183.60.215.50) for the second entry from the input file is not created at all. input file has: 83.149.9.216 - - [28/May/2011:16:13:42 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 183.60.215.50 - - [27/Jun/2012:08:00:00 -0400] "GET /scripts/netcat-webserver HTTP/1.1" 200 182 "-" "Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)" checked the same in ElasticSearch and i see only one document in the index created with today's timestamp. So the second entry from teh apache logfile is not indexed at all. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [July 18, 2019, 11:07am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/12 "2019-07-18T11:07:49Z")
I am still unable to find a fix. Can someone please help me out. I believe this is a teeny tiny issue with config which I am not able to understand. ---
**Author:** ![rohithsolomon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohithsolomon/32/48568_2.png) [@rohithsolomon](https://discuss.elastic.co/u/rohithsolomon)\ **Post date:** [July 25, 2019, 8:06am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/13 "2019-07-25T08:06:46Z")
Could you please let me know which team should be contacted for more help? Not sure if this is related to Elastic Team or Logstash team. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2019, 8:06am UTC](https://discuss.elastic.co/t/logstash-put-all-apache-log-entries-into-one-index-with-the-current-date-in-kibana/186939/14 "2019-08-22T08:06:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.