Cannot create pipeline

syashwanthk · August 15, 2017, 6:40am

Hi,
logstash -e 'input { stdin{} } output { elasticsearch{ host =>"localhost:9200" } }'

Please find the complete logs as below while trying to configure logstash:

[2017-08-15T12:03:17,000][DEBUG][logstash.plugins.registry] [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,015][DEBUG][logstash.runner [2017-08-15T12:03:17,031][DEBUG][logstash.agent [2017-08-15T12:03:17,031][DEBUG][logstash.instru [2017-08-15T12:03:17,062][DEBUG][logstash.instru [2017-08-15T12:03:17,078][DEBUG][logstash.instru [2017-08-15T12:03:17,094][ERROR][logstash.agent [2017-08-15T12:03:17,094][DEBUG][logstash.agent [2017-08-15T12:03:17,094][DEBUG][logstash.instru [2017-08-15T12:03:17,094][DEBUG][logstash.agent [2017-08-15T12:03:17,094][DEBUG][logstash.instru [2017-08-15T12:03:17,094][DEBUG][logstash.agent [2017-08-15T12:03:17,109][DEBUG][logstash.instru [2017-08-15T12:03:17,109][DEBUG][logstash.api.service Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x336dcc30 @module_name="fb_apache", @directory="D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/modules/fb_apache/configuration">}
] -------- Logstash Settings (* means modified) ---------
] node.name: "DESKTOP-J4IPO2R"
] path.data: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/data"
] *config.string: "input { stdin{} } output { elasticsearch{ host = } }"
] modules.cli: []
] modules: []
] config.test_and_exit: false
] config.reload.automatic: false
] config.reload.interval: 3
] metric.collect: true
] pipeline.id: "main"
] pipeline.system: false
] pipeline.workers: 8
] pipeline.output.workers: 1
] pipeline.batch.size: 125
] pipeline.batch.delay: 5
] pipeline.unsafe_shutdown: false
] path.plugins: []
] config.debug: false
] *log.level: "trace" (default: "info")
] version: false
] help: false
] log.format: "plain"
] http.host: "127.0.0.1"
] http.port: 9600..9700
] http.environment: "production"
] queue.type: "memory"
] queue.drain: false
] queue.page_capacity: 262144000
] queue.max_bytes: 1073741824
] queue.max_events: 0
] queue.checkpoint.acks: 1024
] queue.checkpoint.writes: 1024
] queue.checkpoint.interval: 1000
] dead_letter_queue.enable: false
] dead_letter_queue.max_bytes: 1073741824
] slowlog.threshold.warn: -1
] slowlog.threshold.info: -1
] slowlog.threshold.debug: -1
] slowlog.threshold.trace: -1
] path.queue: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/data/queue"
] path.dead_letter_queue: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/data/dead_letter_queue"
] path.settings: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/config"
] path.logs: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logs"
] --------------- Logstash Settings -------------------
] Agent: Configuring metric collection
ment.periodicpoller.os] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
ment.periodicpoller.jvm] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
ment.periodicpoller.persistentqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
] Cannot create pipeline {:reason=>"Expected one of #, => at line 1, column 48 (byte 48) after output { elasticsearch{ host ", :backtrace=>["D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/pipeline.rb:59:in initialize'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/pipeline.rb:156:ininitialize'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/agent.rb:286:in create_pipeline'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/agent.rb:95:inregister_pipeline'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:314:in execute'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:209:in run'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "D:\JAVA_INTEGRATIONS\ELK\logstash-5.5.1\lib\bootstrap\environment.rb:71:in `(root)'"]}
] starting agent
ment.periodicpoller.os] PeriodicPoller: Stopping
] Starting puma
ment.periodicpoller.jvm] PeriodicPoller: Stopping
] Trying to start WebServer {:port=>9600}
ment.periodicpoller.persistentqueue] PeriodicPoller: Stopping
] [api-service] start

I appreciate any feedback on my issue also suggestions how to solve the same.

Thanks.

magnusbaeck · August 15, 2017, 7:27am

Since Logstash 2.0, the elasticsearch output doesn't have a host option. It's called hosts. I would've expected a different kind of error because of that, but start by fixing that obvious error.

syashwanthk · August 15, 2017, 7:59am

Thanks for the quick reply. But change in the input did not change the ERROR as below.

Input: logstash -e 'input { stdin{} } output { elasticsearch{ hosts => [ "localhost:9200" ] }'

[2017-08-15T13:27:54,420][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x336dcc30 @module_name="fb_apache", @directory="D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/modules/fb_apache/configuration">}
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] -------- Logstash Settings (* means modified) ---------
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] node.name: "DESKTOP-J4IPO2R"
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] path.data: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/data"
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] *config.string: "input { stdin{} } output { elasticsearch{ hosts = localhost:9200 ] }"
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] modules.cli: []
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] modules: []
[2017-08-15T13:27:54,533][DEBUG][logstash.runner ] config.test_and_exit: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] config.reload.automatic: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] config.reload.interval: 3
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] metric.collect: true
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.id: "main"
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.system: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.workers: 8
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.output.workers: 1
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.batch.size: 125
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.batch.delay: 5
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] path.plugins: []
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] config.debug: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] *log.level: "trace" (default: "info")
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] version: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] help: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] log.format: "plain"
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] http.host: "127.0.0.1"
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] http.port: 9600..9700
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] http.environment: "production"
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.type: "memory"
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.drain: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.page_capacity: 262144000
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.max_bytes: 1073741824
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.max_events: 0
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] dead_letter_queue.enable: false
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] slowlog.threshold.warn: -1
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] slowlog.threshold.info: -1
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] slowlog.threshold.debug: -1
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] slowlog.threshold.trace: -1
[2017-08-15T13:27:54,537][DEBUG][logstash.runner ] path.queue: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/data/queue"
[2017-08-15T13:27:54,541][DEBUG][logstash.runner ] path.dead_letter_queue: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/data/dead_letter_queue"
[2017-08-15T13:27:54,541][DEBUG][logstash.runner ] path.settings: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/config"
[2017-08-15T13:27:54,541][DEBUG][logstash.runner ] path.logs: "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logs"
[2017-08-15T13:27:54,541][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
[2017-08-15T13:27:54,718][DEBUG][logstash.agent ] Agent: Configuring metric collection
[2017-08-15T13:27:54,718][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-08-15T13:27:54,843][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-08-15T13:27:54,874][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2017-08-15T13:27:54,905][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, => at line 1, column 49 (byte 49) after output { elasticsearch{ hosts ", :backtrace=>["D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/pipeline.rb:59:in initialize'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/pipeline.rb:156:ininitialize'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/agent.rb:286:in create_pipeline'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/agent.rb:95:inregister_pipeline'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:314:in execute'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:209:in run'", "D:/JAVA_INTEGRATIONS/ELK/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "D:\JAVA_INTEGRATIONS\ELK\logstash-5.5.1\lib\bootstrap\environment.rb:71:in `(root)'"]}
[2017-08-15T13:27:54,905][DEBUG][logstash.agent ] starting agent
[2017-08-15T13:27:54,921][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-08-15T13:27:54,921][DEBUG][logstash.agent ] Starting puma
[2017-08-15T13:27:54,921][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-08-15T13:27:54,921][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2017-08-15T13:27:54,921][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-08-15T13:27:54,921][DEBUG][logstash.api.service ] [api-service] start

syashwanthk · August 15, 2017, 8:07am

I could also observe that logstash runs good with the command below.

logstash -e 'input { stdin{} } output { stdout{} }'

But unable to export the same to elasticsearch. Please let me know if you want me to update any config on elasticsearch.

Thanks in advance.

Christian_Dahlqvist · August 15, 2017, 8:11am

You are missing a closing curly brace in the example where you see the error.

syashwanthk · August 15, 2017, 8:14am

Apologies. I pasted it wrong. Anyway I was using the right command as below.

logstash -e 'input { stdin{} } output { elasticsearch { hosts => [ "localhost:9200" ] } }'

magnusbaeck · August 15, 2017, 11:02am

Make sure you're using regular straight quotes everywhere. What you've posted here indicates that you're using curly quotes but it's unclear if that's the actual command you've run or if that damage has been inserted somewhere else.

syashwanthk · August 15, 2017, 11:14am

Hi . It works now. Thanks for your time.

I am a beginner working on ELK from the past few days.

I have few questions here.

I see output.elasticsearch: key configured in filebeat.yml. When I am exporting the logs from filebeat to logstash why do we need to provide the elasticsearch host over in filebeat? When I remove that ,filebeat doesn't start.
I tried updating the log file with few more loggers but could not find the same when tried to search them through kibana. Please help on this.

Thanks in advance.

magnusbaeck · August 15, 2017, 11:46am

When I am exporting the logs from filebeat to logstash why do we need to provide the elasticsearch host over in filebeat?

You don't.

When I remove that ,filebeat doesn’t start.

Then you're doing something wrong. Without knowing what you've tried we can't help.

I tried updating the log file with few more loggers

What do you mean?

Whenever you can, copy/paste actual configuration instead of attempting to describe it.

system · September 12, 2017, 11:47am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash start error Logstash	2	3015	April 2, 2018
[ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x5223bdbe Logstash	1	1027	December 11, 2017
Unable to run this command bin/logstash -e "input { http { port => 9200 } } output { stdout { codec => rubydebug} } Logstash	3	417	June 26, 2019
Problem with Logstash 2.4 with elasticsearch output Logstash	6	2732	July 6, 2017
LogStash-Outputs -ElasticSearch - URI is not valid - host is not specified Logstash docker	2	4097	April 6, 2020

Cannot create pipeline

Related topics