Logstash is not creating Indexes - SOLVED

Vishal_Sharma1 · August 14, 2016, 5:25am

i have logstash running and its suddnely stopped creating indexes. I have installed logstash from the repository on Centos7.
service logstash configtest is ok
sudo service logstash start

starts log stash properly. In /var/log/logstash.log i can see {:timestamp=>"2016-08-13T14:51:56.499000+0000", :message=>"Pipeline main started"}

Below is my conf file
input {
file {
path => "/etc/httpd/logs/access_log"
start_position => "beginning"
}
}

filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug}
}

when i do curl -XGET http://localhost:9200/_cat/indices?v | grep logstash
i can only see old ones and not the new one day something like
logstash-2016.08.14 but i do not see any index also there is no error in logs as well.

I do not know what wrong i am doing here it was working fine. Please help

magnusbaeck · August 14, 2016, 2:07pm

Have you looked in the Logstash log files for clues (they're usually in /var/log/logstash)? Is data actually being added to the end of /etc/httpd/logs/access_log?

Vishal_Sharma1 · August 14, 2016, 2:54pm

Sir,
Yes i did check the logs and all it says pipeline started. To answer your other question yes access_logs generating the data as i have few websites and one web based monitoring system running. So basically it should create and index logstash-2016.08.14 for today but there is nothing.
Please help

magnusbaeck · August 14, 2016, 3:00pm

Try increase the logging verbosity with --verbose and see if there are more clues in the log. If that doesn't help, increase it even more with --debug.

Vishal_Sharma1 · August 14, 2016, 3:02pm

so one more thing i am starting logstash from systemd.
sudo service logstash start.
should i start it from bin directory bin/logstash -v -f /etc/logstsh/conf.d/file.conf ?

Vishal_Sharma1 · August 14, 2016, 4:10pm

ok i have just noticed one thing, when i run logstash from command like as user say
tom$ sudo service logstash start

it doesnt generate anything , then if i run it like below
tom$ bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf

it doesnt do anything

however when i log in as root and do this
root$ bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf

it generates logs and also generates index, so this means its a permission problem ?. One more thing is if even root i start as
root$ service logstash start
it doesnt again generate the data in index. Whats the issue in here ? Please help

magnusbaeck · August 14, 2016, 6:34pm

should i start it from bin directory bin/logstash -v -f /etc/logstsh/conf.d/file.conf ?

That can be useful for debugging, but eventually you'll want to run it via systemd.

it generates logs and also generates index, so this means its a permission problem ?

Yes, probably. Does the logstash user have read access to the log files?

is if even root i start as
root$ service logstash start
it doesnt again generate the data in index.

Which user runs service logstash start doesn't affect which user the service runs as.

Vishal_Sharma1 · August 29, 2016, 10:04am

ok here is one more info if i run below
curl -XGET localhost:9200/logstash-*/_field_stats?fields=@timestamp

i get below
{"_shards":{"total":40,"successful":40,"failed":0},"indices":{"_all":{"fields":{"@timestamp":{"max_doc":9114,"doc_count":9114,"density":100,"sum_doc_freq":36456,"sum_total_term_freq":-1,"min_value":1470571575000,"min_value_as_string":"2016-08-07T12:06:15.000Z","max_value":1471190833000,"max_value_as_string":"2016-08-14T16:07:13.000Z"}}}}}

as you can see the index is there only till 14th Aug and its not generating now. At this time logstash / elasticsearch/ kibana all running fine and i cant see any error. Whats wrong in here any more inputs ?.
Should i remove all three and do it again ?

magnusbaeck · August 29, 2016, 11:12am

Is the file input that tails /etc/httpd/logs/access_log your only input? Has the file been updated since Aug 14? Has Logstash been running since then? What's Logstash's current position in that file (according to the sincedb file)?

Vishal_Sharma1 · August 29, 2016, 11:15am

Yes sir the file updates everyday and with lots of log
And yes logstash was running all the time. I am now doing a fresh install of ELK on the same machine removed everything as it's not been working since long. I must have done something wrong somewhere so let's see if a fresh setup will work.
Thank you for your time and effort and i will update you if the new setup is working.

Vishal_Sharma1 · August 29, 2016, 12:18pm

Now even the fresh installation of everything is not creating the index. Arrgggh
My conf file is at /etc/logstash/conf.d/01-webserver.conf

input {

file {

path => "/etc/httpd/logs/access_log"

start_position => "beginning"

}

filter {

if [type] == "apache-access"

{

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

output {

elasticsearch {

hosts => ["localhost:9200"]

}
input {

file {

path => "/etc/httpd/logs/access_log"

start_position => "beginning"

}

filter {

if [type] == "apache-access"

{

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

output {

elasticsearch {

hosts => ["localhost:9200"]

}
input {

file {

path => "/etc/httpd/logs/access_log"

start_position => "beginning"

}

filter {

if [type] == "apache-access"

{

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

output {

elasticsearch {

hosts => ["localhost:9200"]

}
stdout { codec => rubydebug }

}

is there is anything wrong with this it ?

magnusbaeck · August 29, 2016, 12:49pm

Repeating things I've said before that I don't think you've answered:

Try increase the logging verbosity with --verbose and see if there are more clues in the log. If that doesn't help, increase it even more with --debug.
Does the logstash user have read access to the log files?

Vishal_Sharma1 · August 29, 2016, 12:52pm

ok sir let me paste you some more logs in a while To answer your last question i added the logstash user to adm group

sudo gpasswd -a logstash adm

so i guess this part is ok ? . Now let me post few more logs with --verbose and --debug

Vishal_Sharma1 · August 29, 2016, 12:57pm

Here is the output of bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf. This is the only conf file i have for now. Once i run this i have clicked on my web pages to generate the logs and i can see using tail that access logs generating the data.Please see below link for the --verbose output

gist.github.com

https://gist.github.com/anonymous/74fa56e21bd8bb2b3020269967bc8413

Logstash --verbose output

 bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf 
starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 1
Registering file input {:path=>["/etc/httpd/logs/access_log"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/home/vishne0/.sincedb_b04f3cc6f697ac4a0e3b75cdc3664f75", :path=>["/etc/httpd/logs/access_log"], :level=>:info}
Grok patterns path {:paths=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns", "/opt/logstash/patterns/*"], :level=>:info}
Grok loading patterns from file {:path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/linux-syslog", :level=>:info}
Grok loading patterns from file {:path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/nagios", :level=>:info}
Grok loading patterns from file {:path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/mcollective-patterns", :level=>:info}

This file has been truncated. show original

Vishal_Sharma1 · August 29, 2016, 1:05pm

Logstash --debug ouput

gist.github.com

https://gist.github.com/anonymous/59edf7935d87aaddef7b5f6e2827b9c9

Logstash --debug output

{:timestamp=>"2016-08-29T13:02:17.137000+0000", :message=>"Reading config file", :config_file=>"/etc/logstash/conf.d/webserver.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
{:timestamp=>"2016-08-29T13:02:17.404000+0000", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-08-29T13:02:17.433000+0000", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-08-29T13:02:17.453000+0000", :message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-08-29T13:02:17.457000+0000", :message=>"config LogStash::Inputs::File/@path = [\"/etc/httpd/logs/access_log\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-08-29T13:02:17.467000+0000", :message=>"config LogStash::Inputs::File/@start_position = \"beginning\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-08-29T13:02:17.474000+0000", :message=>"config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>\"UTF-8\">", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-08-29T13:02:17.481000+0000", :message=>"config LogStash::Inputs::File/@add_field = {}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-08-29T13:02:17.484000+0000", :message=>"config LogStash::Inputs::File/@stat_interval = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-08-29T13:02:17.486000+0000", :message=>"config LogStash::Inputs::File/@discover_interval = 15", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}

This file has been truncated. show original

magnusbaeck · August 29, 2016, 1:25pm

{:timestamp=>"2016-08-29T13:02:19.163000+0000", :message=>"_globbed_files: /etc/httpd/logs/access_log: glob is: []", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}

This indicates that /etc/httpd/logs/access_log either doesn't exist or that one of the directories leading up to that file isn't accessible to the user Logstash runs as.

Vishal_Sharma1 · August 29, 2016, 1:45pm

ok i got the point in here as when i ran the command
sudo bin/logstash --debug -f /etc/logstash/conf.d/webserver.conf it worked and generated the index. So i have added logstash user to adm group which means it should have read access to the apache log files. Can you please tell me what group i should add logstash user to so that it works ?

magnusbaeck · August 29, 2016, 2:01pm

Can you please tell me what group i should add logstash user to so that it works ?

Well, what are the permissions and ownerships of the files and directories in question?

Vishal_Sharma1 · August 29, 2016, 2:18pm

so for the directory

drwx------ 2 root root 4096 Aug 28 03:33 httpd

and for file

-rw-r--r-- 1 root root 428540 Aug 29 14:16 access_log

magnusbaeck · August 29, 2016, 2:21pm

Then only the root user can access files in the httpd directory. /etc, /etc/httpd, and /etc/httpd/logs all need to be readable and executable to the logstash user.

To access a file in a directory you normally only need it to be executable but since Logstash attempts to resolve wildcards I'd assume that it also requires the read bit to be set even if you're not using any wildcards.