Logstash not working on Windows

Elastic Stack Logstash
1

Sorry if I'm asking a very basic question. I have setup ELK on windows 7. I am trying to get twitter feed using below config file

input {
twitter {
# add your data
consumer_key => ""
consumer_secret =>""
oauth_token => ""
oauth_token_secret => ""
keywords => ["Trump"]
full_tweet => true
}
}
output {
elasticsearch {
hosts => "localhost"
index => "idx_ls"
document_type => "tweet_ls"
}
}

when i ran below command from bin directory of logstash Im getting below message and index is not added

Pasted image802×140 3.71 KB

2

Looks like the twitter input isn't catching anything. That seems unlikely given the keyword, but I haven't used that input myself. To debug, increase the log level by invoking Logstash with --verbose or even --debug and temporarily replace the elasticsearch output with a simple stdout { codec => rubydebug } output so that Logstash just dumps all events to the terminal.

3

Thanks Magnus. I changed output to

output {
stdout { codec => rubydebug }
}

and ran but getting same as before

Also ran logstash --verbose and got below output. Where can I see logs ?

Pasted image819×189 5.45 KB

4

Where's the reference to your config file in the command?

Please do not post screenshots. Use copy/paste instead.

5

WHat I meant is i tried 2 things ..one is logstash --verbose. got ouput as mentioned in screenshot earlier
and other is logstash -f twitter_logstash.conf and got the same output as before (first post). In the config file I used output as you mentioned

output {
stdout { codec => rubydebug }
}

6

Plain logstash --verbose isn't useful. Please show the output of logstash --verbose -f twitter_logstash.conf.

7

Sorry about that. Below are results both with verbose and debug. BTW I checked for keyword in twitter i saw several tweets.

logstash --verbose -f twitter_logstash.conf
io/console not supported; tty will not be manipulated

←[32mstarting agent {:level=>:info}←[0m
←[32mstarting pipeline {:id=>"main", :level=>:info}←[0m
Settings: Default pipeline workers: 8
←[32mStarting twitter tracking {:track=>"pavanmuktasan", :level=>:info}←[0m
←[32mStarting pipeline {:id=>"main", :pipeline_workers=>8, :batch_size=>125, :ba
tch_delay=>5, :max_inflight=>1000, :level=>:info}←[0m
Pipeline main started

logstash --debug twitter_logstash.conf
its printing a huge log but below lines in bold are printing again and again on the terminal. Checked the pipeline.rb file for line 458 method=>"flush" but did not follow

Pipeline main started {:file=>"/Users/akatti/Desktop/Desktop2/Desktop/Personal/E
lastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-
2.3.1-java/lib/logstash/agent.rb", :line=>"465", :method=>"start_pipeline"}

←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/Users/akatti/Desktop/
Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jr
uby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :
method=>"flush"}←[0m

8

In the debug log, is there anything at all mentioning Twitter?

9

Yes many times..its not letting me post the log here (sayin I can send to max 10 users)...Any other way i can send you..Tried to message you, but same isssue.

10

Perhaps you can use some kind of pastebin service? Or try to read the log messages and see if anything stands out?

11

logstash --debug -e 'input { stdin { } } output { stdout {} }'
←[36mPlugin not defined in namespace, checking for plugin file {:type=>"input",
:name=>"stdin", :path=>"logstash/inputs/stdin", :level=>:debug, :file=>"/Users//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/ve
ndor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :li
ne=>"76", :method=>"lookup"}←[0m
←[36mPlugin not defined in namespace, checking for plugin file {:type=>"codec",
:name=>"line", :path=>"logstash/codecs/line", :level=>:debug, :file=>"/Users//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vend
or/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line
=>"76", :method=>"lookup"}←[0m
←[36mconfig LogStash::Codecs::Line/@charset = "UTF-8" {:level=>:debug, :file=>"/
Users//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2
.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/m
ixin.rb", :line=>"153", :method=>"config_init"}←[0m
←[36mconfig LogStash::Codecs::Line/@delimiter = "\n" {:level=>:debug, :file=>"/U
sers//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.
3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mi
xin.rb", :line=>"153", :method=>"config_init"}←[0m
←[36mconfig LogStash::Inputs::Stdin/@codec = <LogStash::Codecs::Line charset=>"U
TF-8", delimiter=>"\n"> {:level=>:debug, :file=>"/Users//Desktop/Desktop2/
Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/g
ems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :metho
d=>"config_init"}←[0m
←[36mconfig LogStash::Inputs::Stdin/@add_field = {} {:level=>:debug, :file=>"/Us
ers//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3
.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mix
in.rb", :line=>"153", :method=>"config_init"}←[0m
←[36mPlugin not defined in namespace, checking for plugin file {:type=>"output",
:name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"/Use
rs//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.
1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb",
:line=>"76", :method=>"lookup"}←[0m
←[32mstarting agent {:level=>:info, :file=>"/Users//Desktop/Desktop2/Deskt
op/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/l
ogstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"207", :method=>"execute"
}←[0m
←[32mstarting pipeline {:id=>"main", :level=>:info, :file=>"/Users//Deskto
p/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/
jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"461", :m
ethod=>"start_pipeline"}←[0m
Settings: Default pipeline workers: 8
←[36mconfig LogStash::Codecs::Line/@charset = "UTF-8" {:level=>:debug, :file=>"/
Users//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2
.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/m
ixin.rb", :line=>"153", :method=>"config_init"}←[0m
←[36mconfig LogStash::Codecs::Line/@delimiter = "\n" {:level=>:debug, :file=>"/U
sers//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.
3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mi
xin.rb", :line=>"153", :method=>"config_init"}←[0m
←[36mconfig LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::Line charset=>
"UTF-8", delimiter=>"\n"> {:level=>:debug, :file=>"/Users//Desktop/Desktop
2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9
/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :met
hod=>"config_init"}←[0m
←[36mconfig LogStash::Outputs::Stdout/@workers = 1 {:level=>:debug, :file=>"/Use
rs//Desktop/Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.
1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixi
n.rb", :line=>"153", :method=>"config_init"}←[0m

12

←[36mWill start workers for output {:worker_count=>1, :class=>LogStash::Outputs:
:Stdout, :level=>:debug, :file=>"/Users//Desktop/Desktop2/Desktop/Personal
/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-cor
e-2.3.1-java/lib/logstash/output_delegator.rb", :line=>"77", :method=>"register"
}←[0m
←[32mStarting pipeline {:id=>"main", :pipeline_workers=>8, :batch_size=>125, :ba
tch_delay=>5, :max_inflight=>1000, :level=>:info, :file=>"/Users//Desktop/
Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jr
uby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"188", :
method=>"start_workers"}←[0m
Pipeline main started {:file=>"/Users//Desktop/Desktop2/Desktop/Personal/E
lastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-
2.3.1-java/lib/logstash/agent.rb", :line=>"465", :method=>"start_pipeline"}
←[36mPushing flush onto pipeline {:level=>:debug, :file=>"/Users//Desktop/
Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m

13

unable to get this working.. Any help will be appreciated

14

Why did you post logs from the execution of logstash --debug -e 'input { stdin { } } output { stdout {} }' rather than logstash --debug -f twitter_logstash.conf'? It's the latter that we're trying to debug.

15

Issue is the same..below lines just keep on repeating irrespective of command

[36mPushing flush onto pipeline {:level=>:debug, :file=>"/Users//Desktop/
Desktop2/Desktop/Personal/Elastic/logstash-2.3.1/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}←[0m

16

If you want help please post the information that's requested.

17

OK sure. I ran logstash --debug -f twitter_logstash.conf

It generated HUGE number of lines on the terminal and I was not able to catch all. I copied text from entire terminal and I could only see few lines being repeated again n again. Below are the lines ..sorry could not get more. Its pointing to 2 files twitter.rb and pipeline.rb

←[33mundefined method filter' for nil:NilClass {:exception=>#<NoMethodError: un defined methodfilter' for nil:NilClass>, :backtrace=>["C:/logstash/vendor/bund
le/jruby/1.9/gems/logstash-input-twitter-2.2.2/lib/logstash/inputs/twitter.rb:12
8:in run'", "C:/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/ lib/logstash/pipeline.rb:342:ininputworker'", "C:/logstash/vendor/bundle/jruby
/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:336:in `start_input'
"], :options=>nil, :level=>:warn, :file=>"/logstash/vendor/bundle/jruby/1.9/gems
/logstash-input-twitter-2.2.2/lib/logstash/inputs/twitter.rb", :line=>"138", :me
thod=>"run"}←[0m

18

Okay, this is weird. I can't explain it. More context might help.

Never mind copy/paste from the terminal. Capture all the output to a file using output redirection and post that file somewhere. Remember that it'll probably contain your Twitter credentials in clear text so you'll want to redact that information first.

19

OK thanks here it is

{:timestamp=>"2016-04-15T00:08:45.027000+0530", :message=>"Reading config file", :config_file=>"C:/logstash/bin/twitter_logstash.conf", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
{:timestamp=>"2016-04-15T00:08:45.097000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"twitter", :path=>"logstash/inputs/twitter", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-15T00:08:45.707000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-15T00:08:45.727000+0530", :message=>"config LogStash::Codecs::Plain/@charset = "UTF-8"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.727000+0530", :message=>"config LogStash::Inputs::Twitter/@consumer_key = """, :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.727000+0530", :message=>"config LogStash::Inputs::Twitter/@consumer_secret = ", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.727000+0530", :message=>"config LogStash::Inputs::Twitter/@oauth_token = "**"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.737000+0530", :message=>"config LogStash::Inputs::Twitter/@oauth_token_secret = ", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.737000+0530", :message=>"config LogStash::Inputs::Twitter/@keywords = ["logstash"]", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.737000+0530", :message=>"config LogStash::Inputs::Twitter/@full_tweet = true", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.737000+0530", :message=>"config LogStash::Inputs::Twitter/@codec = <LogStash::Codecs::Plain charset=>"UTF-8">", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.737000+0530", :message=>"config LogStash::Inputs::Twitter/@add_field = {}", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.737000+0530", :message=>"config LogStash::Inputs::Twitter/@use_samples = false", :level=>:debug,

20

:file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.747000+0530", :message=>"config LogStash::Inputs::Twitter/@use_proxy = false", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.747000+0530", :message=>"config LogStash::Inputs::Twitter/@proxy_address = "127.0.0.1"", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.747000+0530", :message=>"config LogStash::Inputs::Twitter/@proxy_port = 3128", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:45.747000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-15T00:08:45.757000+0530", :message=>"starting agent", :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"207", :method=>"execute"}
{:timestamp=>"2016-04-15T00:08:45.757000+0530", :message=>"starting pipeline", :id=>"main", :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb", :line=>"461", :method=>"start_pipeline"}
{:timestamp=>"2016-04-15T00:08:47.014000+0530", :message=>"Starting twitter tracking", :track=>"logstash", :level=>:info, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.2/lib/logstash/inputs/twitter.rb", :line=>"120", :method=>"run"}
{:timestamp=>"2016-04-15T00:08:47.014000+0530", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"rubydebug", :path=>"logstash/codecs/rubydebug", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/plugin.rb", :line=>"76", :method=>"lookup"}
{:timestamp=>"2016-04-15T00:08:47.054000+0530", :message=>"config LogStash::Codecs::RubyDebug/@metadata = false", :level=>:debug, :file=>"/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
{:timestamp=>"2016-04-15T00:08:47.374000+0530", :message=>"config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug metadata=>false>", :level=>:debug,