Elasticsearch failed to delete unreferenced translog files due to unexpected Python and OpenSSL upgrade. If the issue is related to upgrade it should be infrastructure independent

Hi all,

I got a production Elasticsearch cluster running on Azure.

On 9th Nov, Azure agent started a yum update on OpenSSL and Python libraries. This action crashes one of the index.

Have anyone with Elasticsearch on Azure experience this before ?

I would like to know if Elasticsearch is indexing data and the following libraries has been updated, Will it crash Elasticsearch ?

(Extract from /var/log/yum.log)

Nov 09 10:39:14 Updated: 1:openssl-libs-1.0.2k-8.el7.x86_64
Nov 09 10:39:17 Updated: python-libs-2.7.5-58.el7.x86_64
Nov 09 10:39:18 Updated: openssh-7.4p1-13.el7_4.x86_64
Nov 09 10:39:18 Updated: python-2.7.5-58.el7.x86_64
Nov 09 10:39:18 Updated: libcurl-7.29.0-42.el7.x86_64
Nov 09 10:39:18 Updated: curl-7.29.0-42.el7.x86_64
Nov 09 10:39:19 Updated: python-devel-2.7.5-58.el7.x86_64
Nov 09 10:39:19 Updated: openssh-clients-7.4p1-13.el7_4.x86_64
Nov 09 10:39:20 Updated: openssh-server-7.4p1-13.el7_4.x86_64
Nov 09 10:39:21 Updated: 1:openssl-1.0.2k-8.el7.x86_64
Nov 09 10:39:22 Updated: 1:openssl-devel-1.0.2k-8.el7.x86_64
Nov 09 10:39:23 Updated: wget-1.14-15.el7_4.1.x86_64

(extract from /var/log/messages)

Nov 9 10:37:45 es0 python: 2017/11/09 10:37:45.027369 INFO Event: name=WALinuxAgent, op=HeartBeat, message=, duration=0
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.252548 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Target handler state: enabled
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.254394 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] [Enable] current handler state is: notinstalled
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.434750 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Download, message=Download succeeded, duration=0
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.435728 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Initialize extension directory
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.440453 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Update settings file: 0.settings
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.441408 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Disable extension [bin/custom-script-shim disable]
Nov 9 10:38:34 es0 python: 2017/11/09 10:38:34.446451 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Disable, message=Launch command succeeded: bin/custom-script-shim disable, duration=0
Nov 9 10:38:34 es0 python: 2017/11/09 10:38:34.448416 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Copy status files from old plugin to new
Nov 9 10:38:34 es0 python: 2017/11/09 10:38:34.448956 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Update extension [bin/custom-script-shim update]
Nov 9 10:38:35 es0 python: 2017/11/09 10:38:35.460504 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Update, message=Launch command succeeded: bin/custom-script-shim update, duration=0
Nov 9 10:38:35 es0 python: 2017/11/09 10:38:35.462176 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Uninstall extension [bin/custom-script-shim uninstall]
Nov 9 10:38:36 es0 python: 2017/11/09 10:38:36.480447 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=UnInstall, message=Launch command succeeded: bin/custom-script-shim uninstall, duration=0
Nov 9 10:38:36 es0 python: 2017/11/09 10:38:36.481668 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Remove extension handler directory: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.4
Nov 9 10:38:36 es0 python: 2017/11/09 10:38:36.483629 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Install extension [bin/custom-script-shim install]
Nov 9 10:38:37 es0 python: 2017/11/09 10:38:37.524445 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Install, message=Launch command succeeded: bin/custom-script-shim install, duration=0
Nov 9 10:38:37 es0 python: 2017/11/09 10:38:37.530731 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Enable extension [bin/custom-script-shim enable]
Nov 9 10:38:38 es0 python: 2017/11/09 10:38:38.556536 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Enable, message=Launch command succeeded: bin/custom-script-shim enable, duration=0
Nov 9 10:38:39 es0 kernel: blk_update_request: I/O error, dev fd0, sector 0
Nov 9 10:38:39 es0 elasticsearch: [2017-11-09 10:38:39,253][WARN ][index.translog ] [es0.----.prod] [xxxx-yyy-7-1_index_2017.11.09][1] failed to delete unreferenced translog files
Nov 9 10:38:39 es0 elasticsearch: java.nio.file.NoSuchFileException: /data3/xxx/nodes/0/indices/xxxx-yyyy-7-1_index_2017.11.09/1/translog
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
Nov 9 10:38:39 es0 elasticsearch: at java.nio.file.Files.newDirectoryStream(Files.java:457)

Regards,
David

This has nothing to do with the upgrades.

This is the problem, the hardware is failing.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.