Reindexing failed with TAG Mismatch Exception

awenest · June 13, 2023, 8:29am

I have increased the socket timeout to 15 minutes since I am trying to reindex ~ 40k records.

But within 6 minutes I get below exception :

Caused by: org.springframework.dao.DataAccessResourceFailureException: Tag mismatch!; nested exception is java.lang.RuntimeException: Tag mismatch!
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:94)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:601)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:584)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.reindex(ElasticsearchRestTemplate.java:303)


2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] Caused by: java.lang.RuntimeException: Tag mismatch!
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:599)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] ... 126 common frames omitted
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] Caused by: java.io.IOException: Tag mismatch!
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:937)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.elasticsearch.client.RestHighLevelClient.reindex(RestHighLevelClient.java:650)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$reindex$9(ElasticsearchRestTemplate.java:304)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:582)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] ... 125 common frames omitted
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] Caused by: javax.net.ssl.SSLException: Tag mismatch!
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:368)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:306)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:688)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:643)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:461)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:440)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:279)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.nio.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:505)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:548)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] ... 1 common frames omitted
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1629)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
2023-06-08T21:10:06.346+05:30 [APP/PROC/WEB/0] [OUT] ... 16 common frames omitte

swallez · June 15, 2023, 8:37am

The root cause is Caused by: javax.crypto.AEADBadTagException: Tag mismatch! which indicates an issue in the TLS/SSL layer, out of the realm of the Java client and Spring Data.

Googling it shows some issues in some TLS 1.3 implementations. Failure with javax.crypto.AEADBadTagException due to a defect in Java 8 TLSv1.3 implementation

You should look at the JVM version, its configuration, or maybe the SSL setup of your Elasticsearch deployment.

awenest · July 7, 2023, 5:23am

I upgraded to JDK 17.0.7 and the issue still exists

system · August 4, 2023, 5:24am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.