Hi guys.
One of our UCs asks for outputting data out of Kafka Connector sink to Elastic Search (Logstash is bypassed).
Now Elastic 6.6 needs SSL to be enabled (which is ok as it should be there for other security considerations).
However this does not work, Kafka Connect does not seem to speak SSL at the moment.
Moreover, here is a quote from their site: https://docs.confluent.io/3.1.2/connect/connect-elasticsearch/docs/elasticsearch_connector.html#security
Security
The Elasticsearch connector can read data from secure Kafka by following the instructions in the Connect security documentation. The functionality to write data to a secured Elasticsearch instance is not yet implemented.
Is this known to the ES team ? While not exactly your problem it impacts an important interface of your product, maybe you know more on it.
Any hint appreciated.
I can totally understand your frustration here but what @ikakavas wrote is the truth. If a 3rd party software does not support a mandatory feature, then you need to ask this 3rd party to add it.
The functionality to write data to a secured Elasticsearch instance is not yet implemented.
The Elasticsearch connector can write data to a secure Elasticsearch cluster that supports basic authentication by setting the connection.username and connection.password configuration properties.
Do you have a chance to test it? I'm not sure if it supports https though. But for sure the right answer is to be found in Kafka mailing list. I don't think there is a way to disable TLS once security is activated on an elasticsearch cluster running 6.6+...
This is not a good way to respond to someone that was only trying to help you, especially on a community forum for an open source project, but frankly ever. Posting confrontationally like this establishes you as an adversary rather than a collaborator, is an ineffective way to make a point, and contributes to building a poor reputation for yourself.
I say this hoping that it will have a positive impact on you but even if not I state it for the benefit of the entire community that we expect civil discourse here.
However that does not change the essence of the matter. You say , quote: "... to respond to someone that was only trying to help you ...". What kind of help are you meaning here ? The answer received simply repeated what myself have already indicated: that Connect does not support out-of-the-box SSL. I had obviously asked for a work-around, if any, based on the assumption that your team has got a lot of experience with that.
Look at the answer of David above, who does make an effort to analyze the situation (really thx for that), admitting of course that the problem lies with Kafka.
So perhaps a nice answer from yourself, apart from putting me to the wall (which is ok) would have also to add something really technically useful. Which you did not. How about, in the future, 1 sentence dealing with moral issues + 1 dealing with technical issues ?
Anyway, thanks for your remarks, they do help me, perhaps mine will also help you and your team.
And really thanks for your balanced answer (again, I do admit my high tone was wrong, still I also do maintain a "commercial" answer is not really helpful in a forum).
but to the point: indeed we do not want to disable TLS. If not for other reason then because we need it to activate SAML authentication for other applications.
So basic authentication is not enough overall.
And anyway, in the end we did find the solution: manually add the server's certificate into client java's "cacerts" keystore. Sure enough, not so convenient and reliable as a simple entry into a dedicated configuration file but for now it worked fine.
In that context, the "help" was the confirmation that your statement is true (Connect does not support out of the box SSL) and the suggestion that the quickest way to resolution for you, would probably be an enhancement request with the maintainers of the software component that was lacking the feature/functionality you need. It might have been short and apologies if the length of it made it sound dismissive, but we try to go through and help with as many topics as possible and sometimes, short answers are what we come up with No matter how much longer I could have explained it though, the cause/analysis/suggestion would remain exactly the same.
You can retain the right to think this is "totally useless" and "unprofessional" as I do retain the right to think otherwise and definitely know that this was neither the intent nor the goal of that message. I'm glad you have found a workaround for the issue and hope this helps others facing the same limitation.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
No matter how much longer I could have explained it though, the cause/analysis/suggestion would remain exactly the same.