Logstash is not creating Indexes - SOLVED

Elastic Stack Logstash
1

i have logstash running and its suddnely stopped creating indexes. I have installed logstash from the repository on Centos7.
service logstash configtest is ok
sudo service logstash start

starts log stash properly. In /var/log/logstash.log i can see {:timestamp=>"2016-08-13T14:51:56.499000+0000", :message=>"Pipeline main started"}

Below is my conf file
input {
file {
path => "/etc/httpd/logs/access_log"
start_position => "beginning"
}
}

filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug}
}

when i do curl -XGET http://localhost:9200/_cat/indices?v | grep logstash
i can only see old ones and not the new one day something like
logstash-2016.08.14 but i do not see any index also there is no error in logs as well.

I do not know what wrong i am doing here it was working fine. Please help

1 Like
2

Have you looked in the Logstash log files for clues (they're usually in /var/log/logstash)? Is data actually being added to the end of /etc/httpd/logs/access_log?

3

Sir,
Yes i did check the logs and all it says pipeline started. To answer your other question yes access_logs generating the data as i have few websites and one web based monitoring system running. So basically it should create and index logstash-2016.08.14 for today but there is nothing.
Please help

4

Try increase the logging verbosity with --verbose and see if there are more clues in the log. If that doesn't help, increase it even more with --debug.

5

so one more thing i am starting logstash from systemd.
sudo service logstash start.
should i start it from bin directory bin/logstash -v -f /etc/logstsh/conf.d/file.conf ?

6

ok i have just noticed one thing, when i run logstash from command like as user say
tom$ sudo service logstash start

it doesnt generate anything , then if i run it like below
tom$ bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf

it doesnt do anything

however when i log in as root and do this
root$ bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf

it generates logs and also generates index, so this means its a permission problem ?. One more thing is if even root i start as
root$ service logstash start
it doesnt again generate the data in index. Whats the issue in here ? Please help

7

should i start it from bin directory bin/logstash -v -f /etc/logstsh/conf.d/file.conf ?

That can be useful for debugging, but eventually you'll want to run it via systemd.

it generates logs and also generates index, so this means its a permission problem ?

Yes, probably. Does the logstash user have read access to the log files?

is if even root i start as
root$ service logstash start
it doesnt again generate the data in index.

Which user runs service logstash start doesn't affect which user the service runs as.

8

ok here is one more info if i run below
curl -XGET localhost:9200/logstash-*/_field_stats?fields=@timestamp

i get below
{"_shards":{"total":40,"successful":40,"failed":0},"indices":{"_all":{"fields":{"@timestamp":{"max_doc":9114,"doc_count":9114,"density":100,"sum_doc_freq":36456,"sum_total_term_freq":-1,"min_value":1470571575000,"min_value_as_string":"2016-08-07T12:06:15.000Z","max_value":1471190833000,"max_value_as_string":"2016-08-14T16:07:13.000Z"}}}}}

as you can see the index is there only till 14th Aug and its not generating now. At this time logstash / elasticsearch/ kibana all running fine and i cant see any error. Whats wrong in here any more inputs ?.
Should i remove all three and do it again ?

9

Is the file input that tails /etc/httpd/logs/access_log your only input? Has the file been updated since Aug 14? Has Logstash been running since then? What's Logstash's current position in that file (according to the sincedb file)?

10

Yes sir the file updates everyday and with lots of log
And yes logstash was running all the time. I am now doing a fresh install of ELK on the same machine removed everything as it's not been working since long. I must have done something wrong somewhere so let's see if a fresh setup will work.
Thank you for your time and effort and i will update you if the new setup is working.

11

Now even the fresh installation of everything is not creating the index. Arrgggh
My conf file is at /etc/logstash/conf.d/01-webserver.conf

input {

file {

path => "/etc/httpd/logs/access_log"

start_position => "beginning"

}

}

filter {

if [type] == "apache-access"

{

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

}

output {

elasticsearch {

hosts => ["localhost:9200"]

}
input {

file {

path => "/etc/httpd/logs/access_log"

start_position => "beginning"

}

}

filter {

if [type] == "apache-access"

{

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

}

output {

elasticsearch {

hosts => ["localhost:9200"]

}
input {

file {

path => "/etc/httpd/logs/access_log"

start_position => "beginning"

}

}

filter {

if [type] == "apache-access"

{

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

}

output {

elasticsearch {

hosts => ["localhost:9200"]

}
stdout { codec => rubydebug }

}

is there is anything wrong with this it ?

12

Repeating things I've said before that I don't think you've answered:

  • Try increase the logging verbosity with --verbose and see if there are more clues in the log. If that doesn't help, increase it even more with --debug.
  • Does the logstash user have read access to the log files?
13

ok sir let me paste you some more logs in a while To answer your last question i added the logstash user to adm group

sudo gpasswd -a logstash adm

so i guess this part is ok ? . Now let me post few more logs with --verbose and --debug

14

Here is the output of bin/logstash --verbose -f /etc/logstash/conf.d/webserver.conf. This is the only conf file i have for now. Once i run this i have clicked on my web pages to generate the logs and i can see using tail that access logs generating the data.Please see below link for the --verbose output

15

Logstash --debug ouput

16 
{:timestamp=>"2016-08-29T13:02:19.163000+0000", :message=>"_globbed_files: /etc/httpd/logs/access_log: glob is: []", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}

This indicates that /etc/httpd/logs/access_log either doesn't exist or that one of the directories leading up to that file isn't accessible to the user Logstash runs as.

17

ok i got the point in here as when i ran the command
sudo bin/logstash --debug -f /etc/logstash/conf.d/webserver.conf it worked and generated the index. So i have added logstash user to adm group which means it should have read access to the apache log files. Can you please tell me what group i should add logstash user to so that it works ?

18

Can you please tell me what group i should add logstash user to so that it works ?

Well, what are the permissions and ownerships of the files and directories in question?

19

so for the directory

drwx------ 2 root root 4096 Aug 28 03:33 httpd

and for file

-rw-r--r-- 1 root root 428540 Aug 29 14:16 access_log

20

Then only the root user can access files in the httpd directory. /etc, /etc/httpd, and /etc/httpd/logs all need to be readable and executable to the logstash user.

To access a file in a directory you normally only need it to be executable but since Logstash attempts to resolve wildcards I'd assume that it also requires the read bit to be set even if you're not using any wildcards.