Curl -k query works but Logstash cannot query

Elastic Stack Logstash
1

When I issue a query via a curl -k command to an https elasticsearch end point, I get the query results, however, using the elasticsearch input plugin in logstash, I cannot retrieve the same data.

curl -k -u xxxx https://server.com:9200/index-*/_search

I get data from this command. From Logstash:

input {
elasticsearch {
hosts => ["server.com:9200"]
index => "index-*"
user => "xxxxx"
password => "xxxxx"
query => '{ "query": { "match_all": {} } }'
}
}
output {
stdout {codec => rubydebug}
}
Logstash returns:

Error: [503]

Network Error
Network Error (tcp_error)

A communication error occurred: ""
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.

With ssl => true, I get:

[2018-01-11T16:11:19,049][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Elasticsearch ssl=>true, hosts=>["server.com:9200"], index=>"index-*", user=>"xxxx", p
assword=>, query=>"{ "query": { "match_all": {} } }", id=>"zzzzzz-1", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_a80bfc29-c019-4c2b-b108-541132f0fee2", enable_metric=>true, charset=>"UTF-8">, size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"]>
Error: certificate verify failed

Any help? Thanks

2

The certificate is failing verification. You should use a certificate on the server that properly verifies. That's also why only curl -k works - it's disabling verification.

You can disable it but that will compromise security.

3

Does it make a difference that I am just stdout the retrieval? This a proof
of concept we just want to make sure we can hit and retrieve from the
target end point.

4

Only you can make security decisions for your own environment.

5

The question is since I am doing a stdout and not an output to elastic, why
am I getting a cert problem. Can i tell logstash, as with the curl -k to
ignore the insecure connection while we prove concept.

6

You've told logstash to connect to ES for input - that's where the verification is failing. It isn't getting anywhere near the output yet.

Perfect! That's the security analysis: "this is a proof of concept only, disabling verification is OK." You can follow the link I posted above to see what to change to disable cert verification: Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

7

To sum up here and please correct me if I have come to the wrong conclusion. The Logstash Ellasticsearch INPUT plugin has an option of turning off SSL on the input request, however, it does not have the ability to ignore an insecure connection (like the curl -k option). So our solution is that we will have to request and install ssl certs on the development machines in order to complete the proof of concept project of extracting parts of a larder index from the divisional cluster to be re-indexed into our department cluster.

8

Ah, I see - I thought I linked to the logstash-input-elastic plugin docs, I was linking to the logstash-output-elasticsearch docs.

As the input plugin doesn't have the ability to disable ssl cert verification, you can set the ca_cert option to a file with the self-signed certificate and it should verify. Or you can add the ability to disable verification to the plugin.

9

Yes. I think we have an accord now. I'll need to get me a cert because I am NOT going to write that code to disable the verification. Thanks for the help.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.