# Kafka Input - avro https schema\_registry\_url: unable to find valid certification path to requested target **URL:** https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438 **Category:** Logstash **Created:** [July 12, 2021, 3:12pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438 "2021-07-12T15:12:24Z") **Posts on this page:** 7 **Page:** 1
### Author: ![Malec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malec/32/46127_2.png) [@Malec](https://discuss.elastic.co/u/Malec) #### Post date: [July 12, 2021, 3:12pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/1 "2021-07-12T15:12:24Z")
Hi friends, Logstash version is 7.11.1 I am having trouble consuming messages from a kafka that uses confluent avro. I can't seem to connect to an https schema registry. Here is my config : ```auto input { kafka { bootstrap_servers => "mybroker.kafka:9094" group_id => "my_group" client_id => "my_client" topics => "mytopic" schema_registry_url => "https://my.registry:8081/" schema_registry_key => "myuser" schema_registry_secret => "mypassword" codec => "json" decorate_events => true auto_offset_reset => "earliest" jaas_path => "/path/to/kafka-jaas-input.conf" sasl_mechanism => "SCRAM-SHA-512" security_protocol => "SASL_SSL" ssl_truststore_location => "/path/to/certs/my.truststore.jks" ssl_truststore_password => "REDACTED" key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer" } } ``` I get the following error message : > Pipeline error {:pipeline\_id=\>"main", :exception=\>#\ I have tried with a schema\_registry\_url that does not use https and it works. Kafka uses SASL\_SSL and my secured connection works. Both Kafka and the registry have the same CA from what I can tell. My hypothesis is that for some reason, the registry http connection does not inherit the truststore information. From my very limited understanding of java, it's not the same class, so that would make sense. Is there any way to either: 1. inherit the truststore from the kafka class 2. Bypass SSL verification altogether for the schema registry. I might be completely wrong and it might just be a cert issue but I don't know how I can verify if it could be the problem or not. Thank you. ---
### Author: ![Malec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malec/32/46127_2.png) [@Malec](https://discuss.elastic.co/u/Malec) #### Post date: [July 13, 2021, 4:05pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/2 "2021-07-13T16:05:20Z")
I decided to use HaProxy as an SSL termination to bypass the security problem. I then was able to get to the following error message : > org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition -0 at offset 7104. If needed, please seek past the record to continue consumption. > Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro unknown schema for id 100769 > Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unauthorized; error code: 401 So my investigations have led me to this topic on SO: [https://stackoverflow.com/questions/67586950/can-not-connect-logstash-to-confluent-schema-registry](https://stackoverflow.com/questions/67586950/can-not-connect-logstash-to-confluent-schema-registry) Which is exactly what my problem is. I tried adding the auth header into my HaProxy Configuration and it worked! To anyone trying to consume avro with a schema registry, just know that as of today, logstash is incapable of managing HTTPS with a truststore or basic authentication. The SO post mentions using the forked codec available [here](https://github.com/revpoint/logstash-codec-avro_schema_registry) I will try it and see if it fixes my problems. For reference, here is my Haproxy configuration used. ```auto frontend registry mode http bind :80 default_backend servers backend servers mode http http-request set-header Authorization Basic\ server srv1 my.registry:8081 ssl verify none #Don't do this in production ``` I believe this deserves raising an issue on the kafka input plugin's github. Edit : [here it is](https://stackoverflow.com/questions/67586950/can-not-connect-logstash-to-confluent-schema-registry) ---
### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger) #### Post date: [July 13, 2021, 5:20pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/3 "2021-07-13T17:20:20Z")
It is unclear to me whether [this](https://github.com/logstash-plugins/logstash-integration-kafka/issues/95) issue/PR from last week addresses this. ---
### Author: ![Malec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malec/32/46127_2.png) [@Malec](https://discuss.elastic.co/u/Malec) #### Post date: [July 15, 2021, 8:40am UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/4 "2021-07-15T08:40:17Z")
Hi Badger, Yes, that seems to be a likely fix for my issue. This is what I was also told on Github. I will update the plugin on my logstash 7.11.1 and see if it works. If it doesn't I'll repeat the plugin upgrade on 7.13.3 (The released version of logstash-integration-kafka for 7.13.3 seems too old to incorporate that fix). Thank you. ---
### Author: ![Malec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malec/32/46127_2.png) [@Malec](https://discuss.elastic.co/u/Malec) #### Post date: [July 15, 2021, 2:57pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/5 "2021-07-15T14:57:11Z")
From what I can tell, we're halfway there. Basic auth now works with logstash-integration-kafka v10.8.1 (latest as of today) but it still won't work while using https. I no longer need to pass the basic auth info directly in haproxy but I still need to use it to terminate TLS encryption and send it over HTTP. [New issue on github](https://github.com/logstash-plugins/logstash-integration-kafka/issues/101) ---
### Author: ![Malec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malec/32/46127_2.png) [@Malec](https://discuss.elastic.co/u/Malec) #### Post date: [July 26, 2021, 3:09pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/6 "2021-07-26T15:09:48Z")
Can't edit my previous post anymore, so for those of you trying to use https with avro schema\_registry\_url please know that it is most likely not working as of V 10.8.1 of the logstash-integration-kafka plugin. I am trying to get this reviewed and tested by someone else so we can confirm 100% that this is indeed a bug in the plugin. If you believe you can help by setting up an avro schema-registry using https and confirming you have the same problem, it would be greaty appreciated. I am marking the previous post as the solution for the problem and I invite anyone in the future to take a look at the github issue to see how things have evolved. Thank you. ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [August 23, 2021, 3:10pm UTC](https://discuss.elastic.co/t/kafka-input-avro-https-schema-registry-url-unable-to-find-valid-certification-path-to-requested-target/278438/7 "2021-08-23T15:10:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.