"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}

Elastic Stack Elasticsearch
1

Hi. I'm trying to follow the "Parsing Logs with Logstash" (Tutorial), and I am having trouble when I try to connect my pipeline to Elasticsearch. My first-pipeline.conf file looks like this:

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

In the terminal where Logstash is running, I'm seeing this error printed to the console:

[2023-02-18T16:09:15,351][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}

My Elasticsearch is running in a separate terminal, but I'm not even able to access it when I try to visit http://localhost:9200 in my browser.

I seem to only be able to connect to Elasticsearch via Kibana, which asks for the "elastic" username and a password that was generated by Elasticsearch when I first ran it.

Any help on trying to troubleshoot why I am unable to connect to Elasticsearch would be greatly appreciated. Thanks!

1 Like
2

How did you do install elasticsearch?

Most likely you did not bind it to the network.

Please share your elasticsearch.yml

you can set

network.host : 0.0.0.0

But then be advised it will be available on your network.

1 Like
3

I downloaded elasticsearch from here: Download Elasticsearch | Elastic

Then I ran the command bin/elasticsearch from within the root directory.

Here is my elasticsearch.yml file:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
# network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 18-02-2023 14:24:41
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["MacBook-Pro-2.local"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

I did try setting network.host : 0.0.0.0 , but that did not work either.

4

Right so elasticsearch is running on HTTPS not HTTP

You have enabled default security.

This also means elasticsearch is using self signed certs ...are you familiar with them?

So first try curl to see if you have access to elasticsearch

curl -k -u elastic https://<ip>:9200

Then in your logstash

hosts => [ "https://localhost:9200" ]

ssl_certificate_verification => false
2 Likes
5

Okay, no I'm not familiar with self signed certs.

So, I was able to connect to elasticsearch using the curl command you provided.

However, when I change the config file in logstash, which now looks like this:

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    elasticsearch {
        hosts => [ "https://localhost:9200" ]

        ssl_certificate_verification => false
    }
}

I'm getting the following warning/error printed to the console:

[2023-02-19T10:08:26,322][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://localhost:9200/'"}
6

Is logstash and Elasticsearch on the same server? I think you said no.

Apologies I used localhost as an example I was showing https

If not you need to put the IP address if the elasticsearch sever ...

hosts => [ "https://<ipelastic search>:9200" ]

7

Yes, logstash and elasticsearch are both running on my local machine. So, I think that localhost is correct in this case.

8

That 401 means authentication error

You need to put the user name and password in too.

user => elastic
password => <yourpassword>

1 Like
9

Okay, so I modified the logstash config file to include the user and password. It now looks like something like this:

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    elasticsearch {
        hosts => [ "https://localhost:9200" ]

        ssl_certificate_verification => false

        user => elastic

        password => <mypassword>
    }
}

I've substituted <mypassword> with the actual password in the actual file.

However, now I'm getting the following error:

[2023-02-19T14:18:14,605][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_] at line 19, column 22 (byte 298) after output {\n    elasticsearch {\n        hosts => [ \"https://localhost:9200\" ]\n\n\tssl_certificate_verification => false\n\n        user => elastic\n\n        password => s", :backtrace=>["/Users/matthewjohnston/launch_school/elk-stack/logstash-8.6.2/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:234:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:168:in `initialize'", "/Users/matthewjohnston/launch_school/elk-stack/logstash-8.6.2/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/Users/matthewjohnston/launch_school/elk-stack/logstash-8.6.2/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/Users/matthewjohnston/launch_school/elk-stack/logstash-8.6.2/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}

Based on this part of the error:

Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_] at line 19, column 22 (byte 298) after output {\n    elasticsearch {\n        hosts => [ \"https://localhost:9200\" ]\n\n\tssl_certificate_verification => false\n\n        user => elastic\n\n        password => s",

I think it's having trouble with the fact that the password contains an asterisk, *, which follows after the s. That asterisk occurs at line 19, column 22 of the config file, which I think is what the error is complaining about.

10

Assume you are not actually putting in the < >

That means insert your info

Put the Password in quotes

Look at the example here

      user => "elastic"
      password => "secret"

All string values should be enclosed quotes

1 Like
11

All seems to be working now—thank you so much for your help!!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.