Logstash is not start as expected

asifalis · December 5, 2015, 7:30am

Hi Expert,

I installed Kibana 4.2 and Elasticsearch 2.1 and logstash 2.0,
Kibana and elasticsearch working fine but logstash not working, the logstash.log

[root@centos logstash]# tailf /var/log/logstash/logstash.log
{:timestamp=>"2015-12-05T01:54:36.987000+0500", :message=>"The error reported is: \n Couldn't find any input plugin named 'lumberjeck'. Are you sure this is correct? Trying to load the lumberjeck input plugin resulted in this error: no such file to load -- logstash/inputs/lumberjeck"}

while lumberjeck is located at required location.

[root@centos logstash]# /opt/logstash/bin/plugin list
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-input-zeromq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-graphite
logstash-output-hipchat
logstash-output-http
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-lumberjack
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-xmpp
logstash-output-zeromq
logstash-patterns-core

Hope Expert will help to resolve this issue.

Asif

magnusbaeck · December 5, 2015, 4:26pm

The plugin is named lumberjack, not lumberjeck.

asifalis · December 7, 2015, 10:54pm

Thank you , I was type wrong package name in configuration, it is working fine now

asifalis · December 8, 2015, 12:08am

lumberjact is resolved but now logstash log showing following entries !!!

[root@centos logstash]# more logstash.log
{:timestamp=>"2015-12-08T04:54:20.200000+0500", :message=>"Failed to install template: connect timed out", :level=>:error}
{:timestamp=>"2015-12-08T04:54:56.894000+0500", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}

any solution to resolve that one issue please. still no data shown in kibana console ,
my configuration file is

input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}]? %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri {}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { hosts => ["192.168.16:39"] }
stdout { codec => rubydebug }
}

vtst2412 · December 8, 2015, 1:02am

Logstash can't connect to ES. You have a typo in your config.

asifalis · December 8, 2015, 3:53pm

but I verified the conf file which is seem ok. following is my config result

[root@centos bin]# ./logstash -f /etc/logstash/conf.d/01-logstash-initial.conf -t
Configuration OK

mturonne · December 8, 2015, 3:59pm

Hi ! Quick answer here : the IP address you specified to connect to ES is not complete. Are you also sure about the tcp port ?

asifalis · December 8, 2015, 4:02pm

ok, It mean i add 9200 port like that one

elasticsearch { hosts => "192.168.16:39:9200" }

asifalis · December 8, 2015, 4:05pm

I add the port and testes but no luck ,

[root@centos bin]# ./logstash -f /etc/logstash/conf.d/30-elasticsearch-output.conf
Default settings used: Filter workers: 2
The error reported is:
the scheme http does not accept registry part: 192.168.16:39:9200 (or bad hostname?)
[root@centos bin]# ./logstash -f /etc/logstash/conf.d/01-logstash-initial.conf
Default settings used: Filter workers: 2
The error reported is:
Address already in use - bind - Address already in usee

mturonne · December 8, 2015, 4:30pm

You are using two colons in your host statement.
I assume the IP address of the server that host ES is 192.168.16.39.
Your output must be : elasticsearch { hosts => ["192.168.16.39:9200"] }

vtst2412 · December 8, 2015, 4:34pm

This is the typo.

asifalis · December 8, 2015, 4:49pm

[root@centos logstash]# cd /opt/logstash/bin/
[root@centos bin]# ./logstash -f /etc/logstash/conf.d/01-logstash-initial.conf
Default settings used: Filter workers: 2
The error reported is:
Address already in use - bind - Address already in use
[root@centos bin]# ./logstash -f /etc/logstash/conf.d/30-elasticsearch-output.conf
Default settings used: Filter workers: 2
Logstash startup completed
Logstash shutdown completed
[root@centos bin]# more /var/log/logstash/logstash.log
{:timestamp=>"2015-12-08T21:43:23.254000+0500", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
[root@centos bin]#

vtst2412 · December 8, 2015, 4:53pm

Your input port is being used, probably by a previous logstash instance.

ps -ef | grep logstash

asifalis · December 8, 2015, 4:55pm

[root@centos bin]# ps -ef |grep logstash
logstash 3371 1 12 21:50 pts/0 00:00:31 /usr/java/jdk1.8.0_65/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log

mturonne · December 8, 2015, 4:58pm

kill -9 3371
And then re-launch logstash.

asifalis · December 8, 2015, 4:59pm

I tried many time but no luck

mturonne · December 8, 2015, 5:01pm

OK
As I am not a Linux expert, I will let the other answer you.

vtst2412 · December 8, 2015, 6:02pm

What is failing? Killing the previous logstash instance or starting a new logstash instance?

We will need more than "no luck" to help you. Can you please provide some output? Which port are you using for input?

sudo netstat -nlp | grep <port> should tell you which process is binding to that port