Can't figure out how to get Logstash to talk to Elastic Cloud

Elastic Stack Logstash
1

Hi there... I'm trying to run Logstash on Ubuntu 16.04 and ship some syslog data to Elastic Cloud.

Here's my /etc/logstash/conf.d/logstash.conf file...

input {
  udp {
    host => "172.31.27.139"
    port => 10514
    codec => "json"
    type => "rsyslog"
  }
}

# This is an empty filter block.  You can later add other filters here to further process
# your log lines

filter { }

# This output block will send all events of type "rsyslog" to Elasticsearch at the configured
# host and port into daily indices of the pattern, "rsyslog-YYYY.MM.DD"

output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "https://dec5765995a367f875b67fb7ef00d33c.us-east-1.aws.found.io:9243" ]
    }
  }
}

I've also configured /etc/logstash/logstash.yml so they include my Elasticsearch Cloud ID, elastic username, and password.

No matter what I try, I keep getting connection errors in my logs. I've tried all of the following combinations:

The logs all show different variations of "Attempted to resurrect connection to dead ES instance" and then a java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit.

I've also verified that AWS security groups have full inbound/outbound traffic allowed to this IP.

Anyone have any ideas what I'm doing wrong?

Thanks!

2

If you follow the details here - https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html - you should be good.

If you have and you are still having issues then we might need to see more complete logs.

3

Yes -- that's what I followed to configure logstash.yml.

Here are the logs from the most recent attempt, hope they help!

[2018-02-07T23:13:34,799][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://dec5765995a367f875b67fb7ef00d33c.us-east-1.aws.found.io:9243/, :path=>"/"}
[2018-02-07T23:13:34,922][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://dec5765995a367f875b67fb7ef00d33c.us-east-1.aws.found.io:9243/", :error_type=>LogStash::Outputs::Elasti
cSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://dec5765995a367f875b67fb7ef00d33c.us-east-1.aws.found.io:9243/'"}
[2018-02-07T23:13:35,330][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `<' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2
-java/lib/logstash/outputs/elasticsearch/common.rb:213:in `get_event_type'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:165:in `event_action_params'", "/usr/shar
e/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39:in `event_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/lo
gstash/outputs/elasticsearch/common.rb:34:in `block in multi_receive'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:i
n `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in `multi_receive'", "/usr/share/logstash/logstash-core
/lib/logstash/pipeline.rb:479:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:478:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:430:in `wor
ker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:in `block in start_workers'"]}
[2018-02-07T23:13:36,256][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>java.nio.channels.ClosedSelectorException, :backtrace=>["sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)", "org.jruby.util.io.SelectorPool.put(SelectorPool.java:8
8)", "org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)", "org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)", "org.jruby.RubyIO.select(RubyIO.java:3405)", "org.jruby.RubyIO$INVOKER$s$0$3$select.call(RubyIO$INVOKER$s$0$3$select.g
en)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:743)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.j
ruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.i
r.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:16
5)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jru
by.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)"
, "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMe
thod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.jav
a:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_MET
HOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAnd
Call(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret
(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java
:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"
]}
[2018-02-07T23:13:36,315][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
4

Also, here's the relevant section of /etc/logstash/logstash.yml, with actual key and passwords crossed out. I triple-checked the credentials, but maybe you'll see something that I did incorrectly?

Encryption_%E2%80%94_ubuntu_ip-172-31-27-139___usr_share_logstash_bin_%E2%80%94_ssh_-i_adelia-rsyslog_pem_ubuntu_ec2-34-207-173-110_compute-1_amazonaws_com_%E2%80%94_256%C3%9758

5

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them :slight_smile:

If you can post that as sanitised text it'd help :slight_smile:

6

You got it! So sorry. :slight_smile:

# ------------ Cloud Settings ---------------
# Define Elastic Cloud settings here.
# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
# and it may have an label prefix e.g. staging:dXMtZ...
# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'
cloud.id: rsyslog:<redacted>
#
# Format of cloud.auth is: <user>:<pass>
# This is optional
# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'
# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'
cloud.auth: elastic:<redacted>
7

What version of Logstash do you have installed?

8

6.2.0, freshly downloaded from https://artifacts.elastic.co/downloads/logstash/logstash-6.2.0.deb a few hours ago.

9

This looks correct, but you also need to specify user and password here as I do not think this output can use the cloud_id data in the logstash.yml file. An example can be found here.

1 Like
10

Hi Christian - that did the trick!

Is there any documentation that clarifies when to use the credentials / cloud ID in the logstash.yml file vs. when to put then in conf.d? The instructions on the page that you sent specifically say to try the .yml approach if you're using 6.0 or later, and I don't think I would have even thought to try that without you telling me to do so.

Is it based just on the output type? If so, is there a list anywhere of which outputs support cloud ID and those that don't?

Thanks for getting me going!

11

I think it is explained quite well in the docs Mark linked to:

Cloud ID applies only when a Logstash module is enabled, otherwise specifying Cloud ID has no effect. Cloud ID applies to data that gets sent via the module, to runtime metrics sent via X-Pack monitoring, and to the endpoint used by X-Pack central management features of Logstash, unless explicit overrides to X-Pack settings are specified in logstash.yml.

12

Ah, OK! So modules are different from Logstash pipeline configuration files. The former can use the Cloud ID, the latter can't. I get it now. Sorry -- fairly new to Logstash setup, so I'm still learning some of the terminology. Thanks for all the help!

Is there any way that I can submit suggested edits for this page? https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html. I have a couple of ideas that might make this point clearer up front.

2 Likes
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.