Hi all,
I got a production Elasticsearch cluster running on Azure.
On 9th Nov, Azure agent started a yum update on OpenSSL and Python libraries. This action crashes one of the index.
Have anyone with Elasticsearch on Azure experience this before ?
I would like to know if Elasticsearch is indexing data and the following libraries has been updated, Will it crash Elasticsearch ?
(Extract from /var/log/yum.log)
Nov 09 10:39:14 Updated: 1:openssl-libs-1.0.2k-8.el7.x86_64
Nov 09 10:39:17 Updated: python-libs-2.7.5-58.el7.x86_64
Nov 09 10:39:18 Updated: openssh-7.4p1-13.el7_4.x86_64
Nov 09 10:39:18 Updated: python-2.7.5-58.el7.x86_64
Nov 09 10:39:18 Updated: libcurl-7.29.0-42.el7.x86_64
Nov 09 10:39:18 Updated: curl-7.29.0-42.el7.x86_64
Nov 09 10:39:19 Updated: python-devel-2.7.5-58.el7.x86_64
Nov 09 10:39:19 Updated: openssh-clients-7.4p1-13.el7_4.x86_64
Nov 09 10:39:20 Updated: openssh-server-7.4p1-13.el7_4.x86_64
Nov 09 10:39:21 Updated: 1:openssl-1.0.2k-8.el7.x86_64
Nov 09 10:39:22 Updated: 1:openssl-devel-1.0.2k-8.el7.x86_64
Nov 09 10:39:23 Updated: wget-1.14-15.el7_4.1.x86_64
(extract from /var/log/messages)
Nov 9 10:37:45 es0 python: 2017/11/09 10:37:45.027369 INFO Event: name=WALinuxAgent, op=HeartBeat, message=, duration=0
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.252548 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Target handler state: enabled
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.254394 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] [Enable] current handler state is: notinstalled
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.434750 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Download, message=Download succeeded, duration=0
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.435728 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Initialize extension directory
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.440453 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Update settings file: 0.settings
Nov 9 10:38:33 es0 python: 2017/11/09 10:38:33.441408 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Disable extension [bin/custom-script-shim disable]
Nov 9 10:38:34 es0 python: 2017/11/09 10:38:34.446451 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Disable, message=Launch command succeeded: bin/custom-script-shim disable, duration=0
Nov 9 10:38:34 es0 python: 2017/11/09 10:38:34.448416 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Copy status files from old plugin to new
Nov 9 10:38:34 es0 python: 2017/11/09 10:38:34.448956 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Update extension [bin/custom-script-shim update]
Nov 9 10:38:35 es0 python: 2017/11/09 10:38:35.460504 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Update, message=Launch command succeeded: bin/custom-script-shim update, duration=0
Nov 9 10:38:35 es0 python: 2017/11/09 10:38:35.462176 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Uninstall extension [bin/custom-script-shim uninstall]
Nov 9 10:38:36 es0 python: 2017/11/09 10:38:36.480447 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=UnInstall, message=Launch command succeeded: bin/custom-script-shim uninstall, duration=0
Nov 9 10:38:36 es0 python: 2017/11/09 10:38:36.481668 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Remove extension handler directory: /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-2.0.4
Nov 9 10:38:36 es0 python: 2017/11/09 10:38:36.483629 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Install extension [bin/custom-script-shim install]
Nov 9 10:38:37 es0 python: 2017/11/09 10:38:37.524445 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Install, message=Launch command succeeded: bin/custom-script-shim install, duration=0
Nov 9 10:38:37 es0 python: 2017/11/09 10:38:37.530731 INFO [Microsoft.Azure.Extensions.CustomScript-2.0.4] Enable extension [bin/custom-script-shim enable]
Nov 9 10:38:38 es0 python: 2017/11/09 10:38:38.556536 INFO Event: name=Microsoft.Azure.Extensions.CustomScript, op=Enable, message=Launch command succeeded: bin/custom-script-shim enable, duration=0
Nov 9 10:38:39 es0 kernel: blk_update_request: I/O error, dev fd0, sector 0
Nov 9 10:38:39 es0 elasticsearch: [2017-11-09 10:38:39,253][WARN ][index.translog ] [es0.----.prod] [xxxx-yyy-7-1_index_2017.11.09][1] failed to delete unreferenced translog files
Nov 9 10:38:39 es0 elasticsearch: java.nio.file.NoSuchFileException: /data3/xxx/nodes/0/indices/xxxx-yyyy-7-1_index_2017.11.09/1/translog
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Nov 9 10:38:39 es0 elasticsearch: at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
Nov 9 10:38:39 es0 elasticsearch: at java.nio.file.Files.newDirectoryStream(Files.java:457)
Regards,
David