Logstach 2.3.2 hang while run with .conf file

dhyeem · May 5, 2016, 9:33pm

Dears ,

there is an issue with the logstash i have , I try to run it with the .conf file i created but it hang after pipeline ,

this is one of the .conf i try to run :

`input {
    stdin {}
        file {
                path =>"/run/media/dhyeem/Logstash/cdr/*"
                type => "cucm-cdr"
                start_position => "beginning"
        add_field => { "StandAloneCluster" => "KAAUH_Cluster" }
        }
        file {
                path =>"/run/media/dhyeem/Logstash/cmr*"
                type => "cucm-cmr"
                start_position => "beginning"
        add_field => { "StandAloneCluster" => "KAAUH_Cluster" }
        }

also i tried with the simple one , it's giving the same issue :

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

`
i only see this result :

[root@localhost conf.d]# /opt/logstash/bin/logstash -f ./cucm.conf  -v


starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/run/media/dhyeem/Logstash/cdr/*"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_97e219595e57bde0712cef7e48bace05", :path=>["/run/media/dhyeem/Logstash/cdr/*"], :level=>:info}
Registering file input {:path=>["/run/media/dhyeem/Logstash/cmr*"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_b0d3cd091fb55e4d83add0276a0a91b4", :path=>["/run/media/dhyeem/Logstash/cmr*"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

`

any advice ?

Regards,

dhyeem · May 5, 2016, 9:33pm

here is the logstash.log view :

[root@localhost bin]# sudo tail /var/log/logstash/logstash.log {:timestamp=>"2016-05-05T08:57:15.979000+0300", :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: The settinghostin pluginelasticsearchis obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:87:inconfig_init'", "org/jruby/RubyHash.java:1342:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:71:inconfig_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/outputs/base.rb:63:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/output_delegator.rb:74:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465:in start_pipeline'"], :level=>:error} {:timestamp=>"2016-05-05T08:57:18.982000+0300", :message=>"stopping pipeline", :id=>"main"} {:timestamp=>"2016-05-05T22:52:50.847000+0300", :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: The settinghostin pluginelasticsearchis obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:87:inconfig_init'", "org/jruby/RubyHash.java:1342:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:71:inconfig_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/outputs/base.rb:63:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/output_delegator.rb:74:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465:in start_pipeline'"], :level=>:error} {:timestamp=>"2016-05-05T22:52:53.849000+0300", :message=>"stopping pipeline", :id=>"main"} {:timestamp=>"2016-05-05T23:18:59.906000+0300", :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: The settinghostin pluginelasticsearchis obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:87:inconfig_init'", "org/jruby/RubyHash.java:1342:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:71:inconfig_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/outputs/base.rb:63:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/output_delegator.rb:74:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465:in `start_pipeline'"], :level=>:error}
{:timestamp=>"2016-05-05T23:19:02.928000+0300", :message=>"stopping pipeline", :id=>"main"}
[root@localhost bin]#

Thanks,

warkolm · May 5, 2016, 10:23pm

That means it has started and it is waiting for data, it hasn't hung.

dhyeem · May 6, 2016, 12:23pm

When i check from the command line while this is hanging , it's saying
[root@localhost logstash]# service logstash status
logstash is not running

when i go with the Document in Elasticsearch i see this error :

[root@localhost logstash]# bin/logstash -e /etc/logstash/conf.d/cucm.conf 
fetched an invalid config {:config=>"/etc/logstash/conf.d/cucm.confinput { stdin { type => stdin } }output { stdout { codec => rubydebug } }", :reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ", :level=>:error}

after that

[root@localhost logstash]# bin/logstash -f /etc/logstash/conf.d/cucm.conf -v

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/run/media/dhyeem/Logstash/cdr/*"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_97e219595e57bde0712cef7e48bace05", :path=>["/run/media/dhyeem/Logstash/cdr/*"], :level=>:info}
Registering file input {:path=>["/run/media/dhyeem/Logstash/cmr*"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_b0d3cd091fb55e4d83add0276a0a91b4", :path=>["/run/media/dhyeem/Logstash/cmr*"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

mnhan · May 6, 2016, 1:07pm

check your logstash config. The pipe line keeps stopping due to config errors. use the logstash --configtest to check your configuration for logstash.

dhyeem · May 6, 2016, 1:47pm

Dear Michael ,

this is the test i made :

[dhyeem@localhost logstash]$ bin/logstash --configtest --config /etc/logstash/conf.d/cucm.conf 

Configuration OK

dhyeem · May 6, 2016, 1:48pm

the installation guide i take is this one :

Regards,

dhyeem · May 6, 2016, 3:20pm

I tryied with this example and got this result :

[dhyeem@localhost logstash]$ bin/logstash agent -f /etc/logstash/conf.d/logstash.simple.conf 
Settings: Default pipeline workers: 2
Pipeline main started


good morning
{
       "message" => "good morning",
      "@version" => "1",
    "@timestamp" => "2016-05-06T15:18:34.592Z",
          "host" => "localhost.localdomain"
}

the config file contain this :

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

warkolm · May 6, 2016, 9:31pm

When you run the command manually, the service check won't apply. It's a whole different way of managing the process.

dhyeem · May 21, 2016, 11:00pm

Dear , when i check with logstash if i start the service it will stay 10 second then will shut down , when i check the log i see this error :

{:timestamp=>"2016-05-22T01:58:16.270000+0300", :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:87:in `config_init'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:71:in `config_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/outputs/base.rb:63:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/output_delegator.rb:74:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465:in `start_pipeline'"], :level=>:error}
{:timestamp=>"2016-05-22T01:58:19.275000+0300", :message=>"stopping pipeline", :id=>"main"}

regards

dhyeem · May 21, 2016, 11:29pm

Dear , thanks for all the support , i figure it out in one of your replays , the input plugin was sayig host => localhost , it should be hosts => localhost like below :

output {
    elasticsearch {
        index => "%{type}-%{+dd.MM.YYYY}"

hosts => localhost 
        }

thanks for all the support