Logstash doesn't create index in ES, no errors

Elastic Stack Logstash
1

I'm having trouble getting logstash to talk to ES.

My log files / console output are here:

http://stackoverflow.com/questions/33245721/logstash-cant-create-an-index-in-elasticsearch

In short, ES and Logstash appear to start fine, have no obvious errors in logs and nothing in the error logs, etc. But, my aliases dict is empty and I don't have a logstash index or data in ES. I really have no idea where to look for answers from here.

Help?

2

Logstash's file input tails files by default. If you want it to read files from scratch make sure you set start_position => beginning and clear existing sincedb entries. See the file input documentation for more information.

General advice: Save yourself time by not attempt to go all the way and connect Logstash to ES until you've established that Logstash gets the messages and processes them correctly. Use a stdout { codec => rubydebug } to establish that.

3

Adding that helped, but only after stopping my service instance of logstash and starting logstash manually via bin/logstash -f [conf file]. Now, everything seems to be working but this isn't the "work around" I wanted.

How can I fix my logstash so it works correctly as a service?

4

The reason it worked when you ran Logstash by hand was that sincedb state isn't shared between users so the files were treated as brand new and were processed from the begininning.

Again, if you want to reprocess log files you have to clear the sincedb entries. If you do that it'll work even when you run Logstash as a service.

5

Where is that file located? I've read it's in the home dir but ls -a | less of my home directory revealed nothing.

6

The sincedb files are stored in the home directory of the user running Logstash. If you start Logstash with --verbose or --debug it'll tell you the exact path to the file being used.

7

{:timestamp=>"2015-10-21T08:57:56.694000-0400", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/root/.sincedb_8f309eb34476af59efaabf28f6aac73a", :path=>["/var/log/python_apps/.log", "/var/log/python_apps/.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"120", :method=>"register"}

I'm new to linux, so forgive me if this isn't correct:

cd /root/ ; ll

total 4
-rw-------. 1 root root 1108 Oct 12 12:20 anaconda-ks.cfg

ls -a | less

.
..
anaconda-ks.cfg
.bash_history
.bash_logout
.bash_profile
.bashrc
.cshrc
.pki
.tcshrc
(END)

Where is it?

8

Don't run Logstash as root. Run it as the logstash user (or some other non-privileged user). That's how the RPM and Debian packages set things up for you.

I don't know what's up with the missing sincedb file.

9

I'm on CentOS-7-x86_64-Minimal-1503-01.

So something in my init is making logstash run as root, which is causing problems?

Or are you saying run things as user logstash manually, and ¯_(ツ)_/¯?

10

So something in my init is making logstash run as root, which is causing problems?

I'm pretty sure the RPMs provided by Elastic create a logstash user that the init scripts are supposed to start Logstash as.

11

So look in the init and see if it's not using user logstash?

12

Yes, something like that. And check what user Logstash actually runs as. Obviously, if HOME is /root but Logstash runs as logstash that would explain why no sincedb file is created.

13

/etc/init.d/logstash

LS_USER=logstash
LS_GROUP=logstash

Also, if this file existed on my box, I would have found it with find regardless of what directory, which I didn't. And this doesn't explain why logstash said such existed in the root folder, anyway.