Logstash put all apache log entries into one index with the current date in kibana

Elastic Stack Logstash
1

Using the latest versions of ELK.
Input data:

100.43.83.137 - - [29/May/2014:02:12:13 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
100.43.83.137 - - [29/May/2014:02:12:17 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
100.43.83.137 - - [29/May/2014:02:12:18 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
181.141.250.44 - - [29/May/2014:02:12:26 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com/search?q=www.google.com&client=ms-android-alcatel&hl=es&source=android-launcher-widget&v=133247963&source=lnms&tbm=isch&sa=X&ei=F1sBU7ewM8_HkAen7IH4Aw&ved=0CAgQ_AUoAA" "Mozilla/5.0 (Linux; U; Android 2.3.6; es-us; ALCATEL ONE TOUCH 4010A Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"
187.211.57.202 - - [29/May/2014:02:13:01 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com.mx/search?q=www.google.com&client=ms-android-americamovil-mx&hl=es&source=android-launcher-search&v=141400000&source=lnms&tbm=isch&sa=X&ei=fk8BU5_HMMrayAGT3YGoCQ&ved=0CAkQ_AUoAA&biw=360&bih=615" "Mozilla/5.0 (Linux; U; Android 4.0.4; es-us; ALCATEL ONE TOUCH 5035A Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"

logstash conf file:
#input { stdin { codec => plain } }
input
{
file {
path => "/home/ec2-user/data/testlog"
type => "logs"
start_position => "beginning"
}
}
#output { stdout { codec => rubydebug } }

output { stdout { codec => rubydebug } elasticsearch { hosts => "elkcluster:9200" } }

filter {
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
remove_field => "timestamp"
}
mutate{
convert => { "bytes" => "integer" }
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}

In Kibana,

I see only one index with todays timestamp instead of 5 indices. been trying many combinations but am not able to figure out whats wrong.

On the console, logstash output shows: (trimmed output)

          "timezone" => "America/Mexico_City",
     "country_code3" => "MX",
         "city_name" => "León"
},
      "ident" => "-",
 "@timestamp" => 2014-05-29T07:13:01.000Z,
"httpversion" => "1.1",
   "referrer" => "\"http://www.google.com.mx/search?q=www.google.com&client=ms-android-americamovil-mx&hl=es&source=android-launcher-search&v=141400000&source=lnms&tbm=isch&sa=X&ei=fk8BU5_HMMrayAGT3YGoCQ&ved=0CAkQ_AUoAA&biw=360&bih=615\"",
       "type" => "logs",
       "verb" => "GET",
    "request" => "/images/googledotcom.png",
   "clientip" => "187.211.57.202"
2

Can someone please have a look at this issue.

1 Like
3

Why would you see 5 indices with that configuration? You are using the output plugins defaults which creates one ´logstash-*´prefixed index per day.

4

Hi Chris:

I had to tear down EC2 setup as i had consumed the allowed free-tier resource limit. So in order to replicate the scenario, i did the same on local machine.

logstash conf:

its the same except

input
{
file {
path => "E:/ElasticSearch/logs1"
type => "logs"
start_position => "beginning"
}

}

logs1 file has 2 entries with dates 28th May 2011 and 27th Jun 2012:

83.149.9.216 - - [28/May/2011:16:13:42 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
183.60.215.50 - - [27/Jun/2012:08:00:00 -0400] "GET /scripts/netcat-webserver HTTP/1.1" 200 182 "-" "Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)"

output i see when i run logstash:

     "type" => "logs",
 "@timestamp" => 2011-05-28T21:13:42.000Z,
       "auth" => "-",
       "verb" => "GET",
   "@version" => "1",
   "response" => "200",
   "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
      "bytes" => 203023,
      "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\""

In Kibana i dont see 2 indices. instead there is one with TODAY's timestamp ( logstash-2019.06.28-000001). Its the same issue I faced while running the services on AWS.

6

I am not sure if the configuration I used in apache.conf file is wrong. Its the standard Hello World type example for Logstash and i am pretty sure the grok pattern used is correct as well. Coz the output of logstash on the console correctly identifies the @timestamp but when the data is indexed into ES and read in Kibana, something is happening with the process which is not creating the output as expected.

7

has something changed with the grok pattern I used? I know this isnt a prod issue but we are working on a POC to implement ELK stack. As pasrt of my research i found this simple program not working. Any help would be highly appreciated.

8

I believe this works well in Elastic version 6. Is this some issue with version 7?

9

If your events have @timestamp in 2011 then clearly the grok and date filters are working. You seem to be asserting that an elasticsearch output is putting that into an index named with today's timestamp. If that is the case then the configuration of the output is what we need to see.

10

Here is the output configuration. I haven't used any fancy operations, just printed it on the console and then indexed into ES. Did i miss anything?
output
{
stdout {
codec => rubydebug
}
elasticsearch { }
}

11

There is another issue I just noticed:

input file has 2 entries as shown at the beginning of the post. When i run logstash, I see just one index on console (related to the IP 83.149.9.216). A second index (related to 183.60.215.50) for the second entry from the input file is not created at all.

input file has:

83.149.9.216 - - [28/May/2011:16:13:42 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
183.60.215.50 - - [27/Jun/2012:08:00:00 -0400] "GET /scripts/netcat-webserver HTTP/1.1" 200 182 "-" "Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)"

checked the same in ElasticSearch and i see only one document in the index created with today's timestamp. So the second entry from teh apache logfile is not indexed at all.

12

I am still unable to find a fix. Can someone please help me out. I believe this is a teeny tiny issue with config which I am not able to understand.

13

Could you please let me know which team should be contacted for more help? Not sure if this is related to Elastic Team or Logstash team.

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.