LogStash 2.2 start as a service does not read config file

Hi,

I am running logstash 2.2 on ubuntu 14.0.4.

I SSH into my linux box using a cert and sudo to root user. My config file is under /etc/logstash/conf.d/default.conf.

a) When I run logstash from the command line:
#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/default.conf

I see after ~10 seconds:
Settings: Default pipeline workers: 1
Logstash startup completed

When I send logs to my nginx which LogStash is reading I see an increase in my ElasticSearch entry count via:
#curl 'localhost:9200/_count?pretty=1'

b) When I run logstash as a service:
#service logstash start

It instantly prints:
logstash started

service logstash status says the service is running. But it doesn't seem to read my default.conf file since I see no increase in hits when I run:
#curl 'localhost:9200/_count?pretty=1'

I have tried to fix this via:

  1. changing file owner and group permissions on my default.conf file
  2. adding a logstash.conf file
  3. reading the /etc/init.d/logstash file - no joy
  4. I see no errors in the /var/log/logstash/logstash.err or logstash.log files

So, how do I run logstash as a service and have it read my default.conf file?

Many thanks

Try upping the log level by starting Logstash with --verbose (or even --debug, but then it starts to get hard to identify the really interesting parts). You pass those options by modifying LS_OPTS in /etc/default/logstash. It could be a permission problem with the log files, which Logstash will tell you about in its verbose logs.

can you share with us our config file for logstash?
are you sure about permission for logstash (not only config) but all applicaion folder (ex. /opt/logstash)?
run logstash wit DEBUG=1 or verbose.

Sorry for the delay, it started to work! Now I updated my logstash config file and it stopped working again!

I changed the log level to debug as you suggested and now my log file spits out a lot of info that I have no idea what it's doing!

My config file is under /etc/logstash/conf.d/default.conf

service logstash configtest

  • says config is ok

Where does Logstash look for the config file?

I run logstash as a service as follows:
sudo service logstash start

This results in seeing:
sudo service logstash restart
Killing logstash (pid 25831) with SIGTERM
Waiting logstash (pid 25831) to die...
Waiting logstash (pid 25831) to die...
logstash stopped.
logstash started.

But, the logstash.log says:
{:timestamp=>"2016-04-07T15:19:25.511000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}

So it looks like, this pipeline is what ingests the nginx logs and it's not open???

service logstash status

  • says "logstash is running"

The contents of these folders are all owned by the 'logstash' user and group:
/var/log/logstash
/opt/logstash
/etc/logstash

On startup, my logstash.log file says:

{:timestamp=>"2016-04-07T15:08:21.282000+0000", :message=>"Registering file input", :path=>["/var/log/nginx/web_pixels.log"], :level=>:info}
{:timestamp=>"2016-04-07T15:08:21.299000+0000", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/var/lib/logstash/.$
{:timestamp=>"2016-04-07T15:08:21.825000+0000", :message=>"Using mapping template from", :path=>nil, :level=>:info}
{:timestamp=>"2016-04-07T15:08:22.365000+0000", :message=>"Attempting to install template", :manage_template=>{"template"=>"logstash-*", "settings"=>{"inde$
{:timestamp=>"2016-04-07T15:08:22.750000+0000", :message=>"New Elasticsearch output", :class=>"LogStash::Outputs::ElasticSearch", :hosts=>["127.0.0.1:9200"$
{:timestamp=>"2016-04-07T15:08:23.626000+0000", :message=>"Using geoip database", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-$
{:timestamp=>"2016-04-07T15:08:24.469000+0000", :message=>"Starting pipeline", :id=>"base", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_i$
{:timestamp=>"2016-04-07T15:08:24.519000+0000", :message=>"Pipeline started", :level=>:info}

Please help!

Here's my config file:

input {
        #stdin {}
        file {
                path => "/var/log/nginx/web_pixels.log"
                type => "web"
        }
}

output {
        #stdout { codec => rubydebug }
        elasticsearch {
                hosts => ["127.0.0.1:9200"]
                document_id => "%{[@metadata][doc_id]}"
                #document_type => "%{pr}"
        }
}

filter {
        kv{}
        kv {
                field_split => "&?"
                exclude_keys => ["rv"]
                source => "args"
        }
        date {
                match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
        }

        fingerprint {
                source => [ "message", "type"]
                key => "my-key"
                concatenate_sources => true
                target => "[@metadata][doc_id]"
        }


        geoip {
                source => "clientip"
                #fields => ["city_name", "country_name", "timezone", "country_code2"]
                add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]

        }
        mutate {
                remove_field => [ "message", "args", "countrycode", "city" ]
                convert => [ "[geoip][coordinates]", "float"]
        }

        useragent {
                source => "agent"
                prefix => "ua_"
        }
}

The log makes it seem like it's working. What exactly is not working?

Where does Logstash look for the config file?

The Logstash sysconfig is at /etc/sysconfig/logstash so check what is set for LS_CONF_DIR.
If that file is missing then the default is also kept in the start script at /etc/init.d/logstash.

In both cases they are set to /etc/logstash/conf.d by default though so it should be finding it fine if you haven't changed anything.

...after more investigating Logstash didn't have permission to read my nginx file! I changed the permission of my nginx log folder to worldwide (chmod 777).

It would be nice to see the following entries in the logstash log to show what state it's in:
a) 'permission denied' message in the logstash log (verbose level)
b) pipline open/started (non verbose level, since it shows the pipe is shut at this low level)

Thanks.

Alex,

I tried to locate both of those files but was not able to : /etc/sysconfig/logstash and /etc/init.d/logstash. Any suggestions as to where to find them?

Thanks,

Hi Rinat. You should try to avoid resurrecting old threads!

Those files are installed in the RPM version. Is that the version you used?

Thanks for the correction. I just tried to avoid to create duplicate topics while similar issues were addressed before. I'll avoid resurrecting old ones from now.

I used this :

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

sudo yum install logstash

sudo systemctl start logstash.service

Is this what you are referring to?!

I have the same problem - checked configs and ran the following all to no avail.
sudo systemctl enable logstash.service
^ start logstash.service

When I tail -f /var/log/logstash/logstash-plain.log I do not see the system as starting up at all. Only when I run the manual /bin/logsash with a config file does the service start.

Could my problem be that I once applied, but no longer have the free trial of X-Pack authentication? dropped it like it was hot. And now have no authentication on the Logstash config where there once was a certificate and login?

Thanks