Failed to connect to App Search. Failed to open TCP connection to https:443 (initialize: name or service not known)

joeripeeters · May 26, 2021, 8:23am

I've got a logstash.conf file defining the following:

        input {
      jdbc {
        jdbc_validate_connection => true
        jdbc_driver_library => ""
        jdbc_driver_class => ""
        jdbc_connection_string => "jdbc:mysql://x.mysql.mysql.database.azure.com:3306/x"
        jdbc_user => "xx"
        jdbc_password => "xx"
        schedule => "* * * * *"
        clean_run => false
        statement => "select * from x"
      }
    }
    output {
          elasticsearch {
            hosts => ["https://xxxx.westeurope.azure.elastic-cloud.com:9243"]
            user => "elastic"
            password => "xxx"
            data_stream => "auto"
            index => "xxx_%{[@metadata][index]}"
          }
          elastic_app_search {
              host => "https://xxxx.ent.westeurope.azure.elastic-cloud.com"
              api_key => "private-xxxxx"
              engine => "myengine"
            }
          stdout { codec => rubydebug }
        }

The push to Elasticsearch is working, but I can't manage to push the data to elastic_app_search, I get the following stack trace:

{"level":"ERROR","loggerName":"logstash.javapipeline","timeMillis":1622016095616,"thread":"[main]-pipeline-manager","logEvent":{"message":"Pipeline error","pipeline_id":"main","exception":{"metaClass":{"metaClass":{"metaClass":{"exception":"Failed to connect to App Search. Failed to open TCP connection to https:443 (initialize: name or service not known)","backtrace":["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elastic_app_search-1.1.1/lib/logstash/outputs/elastic_app_search.rb:36:in `register'","org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'","org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'","org/jruby/RubyArray.java:1809:in `each'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"],"pipeline.sources":["/usr/share/logstash/pipeline/logstash.conf"],"thread":"#<Thread:0x6956ec45 run>"}}}}}}
{"level":"INFO","loggerName":"logstash.javapipeline","timeMillis":1622016095627,"thread":"[main]-pipeline-manager","logEvent":{"message":"Pipeline terminated","pipeline.id":"main"}}
{"level":"ERROR","loggerName":"logstash.agent","timeMillis":1622016095654,"thread":"Converge PipelineAction::Create<main>","logEvent":{"message":"Failed to execute action","id":{"metaClass":{"metaClass":{"metaClass":{"id":"main","action_type":{"metaClass":{"metaClass":{"action_type":"LogStash::ConvergeResult::FailedAction","message":"Could not execute action: PipelineAction::Create<main>, action_result: false","backtrace":null}}}}}}}}}
{"level":"TRACE","loggerName":"logstash.agent","timeMillis":1622016095664,"thread":"Agent thread","logEvent":{"message":"Converge results","success":false,"failed_actions":["id: main, action_type: LogStash::PipelineAction::Create, message: Could not execute action: PipelineAction::Create<main>, action_result: false"],"successful_actions":[]}}
{"level":"DEBUG","loggerName":"logstash.instrument.periodicpoller.os","timeMillis":1622016095713,"thread":"LogStash::Runner","logEvent":{"message":"Stopping"}}
{"level":"DEBUG","loggerName":"logstash.instrument.periodicpoller.jvm","timeMillis":1622016095727,"thread":"LogStash::Runner","logEvent":{"message":"Stopping"}}

For reference; this is my dockerfile:

FROM docker.elastic.co/logstash/logstash:7.13.0

# Install necessary logstash filters
# By default logstash-filter-jdbc and streaming is enabled
RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-aggregate
RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-mutate
RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-elastic_app_search

USER logstash
# Copy mysql connector for java to logstash core dir
COPY --chown=logstash:root ./libs/mysql-connector-java-8.0.25.jar /usr/share/logstash/logstash-core/lib/jars/mysql-connector-java.jar

# Copy logstash config files into conf directory
COPY ./logstash.conf /usr/share/logstash/pipeline/logstash.conf
COPY ./logstash.yml /usr/share/logstash/config/logstash.yml

ross.bell · May 26, 2021, 3:17pm

Hey @joeripeeters,

Using the connection info in your config's elastic_app_search entry, are you able to successfully issue API requests to the App Search deploy manually without Logstash (example)?

Ross

joeripeeters · May 26, 2021, 3:34pm

Hi Ross,

Thanks for replying.

yes I can do a POST request using postman to the search endpoint
https://enterprise-search-deployment-xxxx.ent.westeurope.azure.elastic-cloud.com/api/as/v1/engines/xxx/search

I get a response back.

But ofcourse, this is outside the docker container. I will try to do a curl inside the docker container (if that would make a difference)

{
    "meta": {
        "alerts": [],
        "warnings": [],
        "precision": 2,
        "page": {
            "current": 1,
            "total_pages": 0,
            "total_results": 0,
            "size": 10
        },
        "engine": {
            "name": "xxxx",
            "type": "default"
        },
        "request_id": "cju8MU_xRyGfCg7LM6aKcw"
    },
    "results": []
}

ross.bell · May 26, 2021, 3:57pm

Hey @joeripeeters,

Yah that would be my next troubleshooting step. Given the error:

Failed to connect to App Search. Failed to open TCP connection to https:443 (initialize: name or service not known)

I'm not sure this is specific to Logstash or App Search. Instead, this feels like a network visibility issue.

Ross

joeripeeters · May 26, 2021, 7:48pm

Unfortunately the curl command succeeds in the docker container ;-).

Not sure where to look for other information. The error is too verbose.

"initialize: name or service not known" is thrown by one of Ruby's HTTP classes

bash-4.2$ curl -X POST 'https://enterprise-search-deployment-xxx.ent.westeurope.azure.elastic-cloud.com/api/as/v1/engines/xxxxx/search' -H 'Content-Type: application/json' -H 'Authorization: Bearer private-xxxxxxxx' -d '{"query": "everglade"}'
{"meta":{"alerts":[],"warnings":[],"precision":2,"page":{"current":1,"total_pages":0,"total_results":0,"size":10},"engine":{"name":"cultuurnet","type":"default"},"request_id":"DS2EhGT-TOGWt5K5p8Z9OQ"},"results":[]}bas

joeripeeters · May 26, 2021, 8:17pm

Hi @ross.bell
I managed to make it work by specifying the url property instead of host. I am using the SAAS (elastic.co) solution so it doesn't make sense and seems like a bug, since host property is meant for SAAS and url for self-managed.

This works:

  elastic_app_search {
      url => "https://enterprise-search-deployment-xxxx.ent.westeurope.azure.elastic-cloud.com"
      api_key => "private-xxxx"
      engine => "xxxx"
      path => "/api/as/v1/"
    }

{"level":"ERROR","loggerName":"logstash.javapipeline","timeMillis":1622060011726,"thread":"[main]-pipeline-manager","logEvent":{"message":"Pipeline error","pipeline_id":"main","exception":{"metaClass":{"metaClass":{"metaClass":{"exception":"Failed to connect to App Search. Both "url" or "host" can't be set simultaneously. Please specify either "url" (for self-managed) or "host" (for SaaS).","backtrace":["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elastic_app_search-1.1.1/lib/logstash/outputs/elastic_app_search.rb:36:in

ross.bell · May 26, 2021, 8:19pm

Oh, what if you set the host field to enterprise-search-deployment-xxxx.ent.westeurope.azure.elastic-cloud.com (just the hostname, no scheme).

joeripeeters · May 26, 2021, 8:23pm

elastic_app_search {
host => "enterprise-search-deployment-xxx.ent.westeurope.azure.elastic-cloud.com"
api_key => "private-xxx"
engine => "xxx"
}

Fails with: {"exception":"Failed to connect to App Search. hostname \"enterprise-search-deployment-xxx.ent.westeurope.azure.elastic-cloud.com.api.swiftype.com\" does not match the server certificate",

What is the relationship between the host I specify and swifttype.com ? (which differs?)

ross.bell · May 26, 2021, 8:26pm

That's really unexpected to me. Maybe host really is only usable with Swiftype (a similar-but-not-the-same SaaS offering of App Search). Then the way to go for you must be url and no host.

joeripeeters · May 26, 2021, 8:30pm

Indeed counter intuitive since I am solely using elastic.co and using the official documention here:

nothing is mentioned; maybe worth to highlight this to the Logstash support team.

Thanks for your support Ross, greatly appreciated.

Cheers
Joeri

system · June 23, 2021, 8:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.