Master Node Error - Read-only file system

iamkelvinloke · February 28, 2018, 8:38am

In Active Master node. we keep having the logs of below example. This happens to all indices and all files in /var/lib/elasticsearch/nodes/0/indices/...

We are using local disk.
Disk usage is not full.
We tried to use the user "elasticsearch" and we are able to have write permission in /var/lib/elasticsearch
We tried to switch to another Master node, these WARNING logs will then continue to show up in that active Master node.
We tried to check the number of open files, it's lesser than 1000 so will be fine.
We tried to reboot the Master node but it doesn't help.

Elasticsearch: 5.5.1
Nodes:
1 * Load Balancing node
3 * Master nodes
3 * Data & Ingest nodes (Hot)
3 * Data & Ingest nodes (Warm)

The cluster is still running fine as of now, it's still able to do the indexing of new data and able to query old data as well. The only impact is that the status of all indices shown in Monitoring page, will show as "Unknown".

Does anyone of you encounter similar problem before? The problem started 2 days ago, although it causes not much of harm but the Elasticsearch logs keep growing (~100GB a day) and we are not able to check the indices status in Monitoring page.

iamkelvinloke · February 28, 2018, 8:39am

Here is the Warning logs:

[2018-02-28T08:28:56,372][WARN ][o.e.c.a.s.ShardStateAction] [irlpe-logs-02] [filebeat-varnish-prod-2018.02.27][0] received shard failed for shard id [[filebeat-varnish-prod-2018.02.27][0]], allocation id [QTvqMg5tRginaZfQ6hFbQQ], primary term [0], message [failed to create shard], failure [FileSystemException[/var/lib/elasticsearch/nodes/0/indices/sXlcl4-LT569YnALj5KCxA: Read-only file system]]
java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/sXlcl4-LT569YnALj5KCxA: Read-only file system
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_131]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_131]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_131]
at org.elasticsearch.index.store.FsDirectoryService.newDirectory(FsDirectoryService.java:89) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.index.store.Store.(Store.java:159) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.index.IndexService.createShard(IndexService.java:346) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:500) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:148) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:542) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:519) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:204) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Ben96 · March 2, 2018, 12:22pm

Hi,

Looks like you got a Permission Issue on your Indices folder
Elastic couldn't write into the folder mentioned
java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/sXlcl4-LT569YnALj5KCxA: Read-only file system

Could you show me the result of :

"ls -la" or "ls -l" or "ll" for the folder /var/lib/elasticsearch/nodes
and maybe subfolder

Hoang_Ho · March 3, 2018, 10:47pm

Hi,

This is the result of "ll" command:

drwxrwxrwx 3 elasticsearch elasticsearch 4096 Jul 8 2017 ./
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Jul 8 2017 ../
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Feb 28 07:30 0/

drwxrwxrwx 4 elasticsearch elasticsearch 4096 Feb 28 07:30 ./
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Jul 8 2017 ../
drwxrwxrwx 75 elasticsearch elasticsearch 4096 Mar 3 00:57 indices/
-rwxrwxrwx 1 root root 14 Feb 28 04:36 kelvin*
-rwxrwxrwx 1 elasticsearch elasticsearch 0 Jul 8 2017 node.lock*
drwxrwxrwx 2 elasticsearch elasticsearch 4096 Mar 3 00:57 _state/

drwxrwxrwx 75 elasticsearch elasticsearch 4096 Mar 3 00:57 ./
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Feb 28 07:30 ../
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Feb 28 23:57 0AB_id0DS-qtx7xbJhtm_w/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 2RBx7idhT_Sxx86K4phP7g/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 2sRfuMSVSviSkcJqjyAUmQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 3Ivv79LHSvyJ_4NgoBAY3w/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Feb 28 08:18 5QVG-OB5TBeYa7m-2yNwPQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 5sUHyiNrRLG980bMl02dqQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 7AyvJaWbSTyouWyMJCkwog/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 9oC5EYCnRw2pQ8VOjALEvw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 aewNdhk1Tiy_EtA-IRIAuQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 B1cAHha1SNW6wj3nbTi43Q/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Feb 28 23:56 bBDOzSInQCOJsyRgTHq7dA/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 1 07:23 bIIL8ubcQ9Kp6Rw-vJDayw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 BIqWKcntSa2gliWEpB7sHg/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 bsKxbK3lQgu9Y0wxt0Aptw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 BTvBvpJqT9aw05RhNL2GqQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 c74-XnH5Ty6PJdTYq4u2jg/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 cY8eDHEWT9ybVeI_kUaWvA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 D2dfxIq9RmmKAHWBrz_psA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 dV-mW_wdS5iFFiJm_qK4kA/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 2 00:00 eESGiBxZRy2m3bepxYC3vw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 f2rQcm4ORby59ruPBPG4EA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 FhlH014cTpOUZ1k3szYjyg/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 fYw4flbpTBmNvK_dctJSag/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 G7yJuWgBRaOX3Pr0WKiF6w/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 gCLRoff3TziLgbJNtGXuGA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 GJzNrjfJSK2UevlmuEFgUQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 h6Jx-Eo9Q9q6RmneVjiZKw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 HmHpd10BSRSDaFj6eAXEQg/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 ikYNJMopTY-S8ML3TCsvXg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 1 00:00 jo6TLs_fT8m2_YSfwB-jdg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 2 23:57 JrzHsCQrSiyxzr8lV-yjBg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 2 23:56 JzZO01gISBmExWBHPswGag/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 k4n_XMgGSkqr8jkhyDDZyQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 -KGvcfHkRSqxHUpWKHDocQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 L_BxirxwRHeqrJmZ58s9fw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 lrenXn9uT5qyeNssv2lHMQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 m7qFhnO5TmOTOH_Zt-rs2A/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 Nn5cO_xcQ86SIfSfXSXpIQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 nxPHVltmQn-TeD1fmxuvBg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 1 00:00 Nz4iqWroSLSeqgROULq1Mw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 OCIqLjlnSS240UiPPsEG8g/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 oCuuajAsTpymhAui_RYFHA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 Om293LCoTG6Dm6-b7VjauQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 oNaQcg39Rv-cRj67lxUW3g/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 Papu2vGvTPKBTDzYjdREfQ/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 2 00:00 -Pb3T9eSTNG9qZNTLVrDwg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 1 23:57 PfP8Ei6UQ2uKv7EZDWqScQ/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 1 23:56 PHvSGO13Rc-DiPvCauN3GQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 PhY6jv9JR821GxoEluQRRw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 pjcBW41ASAWd1cZVY0L87Q/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 pKAHB6ZPQV64icJxt-LzMg/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 PMJonoYqQWqvkqUx5qDQUQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 Qe814jTwQfCSux2Cl6opMw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 QGu5qIGsTG28FC2Xm6gomA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 RHf7Ru2tQveVTAUy1ZLYcQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 RSSpE1D0SsS6YcdWwSINwA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 S5TNcX26RbOEa5_6yDI13g/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 sXlcl4-LT569YnALj5KCxA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 TCCLjMRoSQKB1TUK0YyrMA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 TjejlhhPSvynOAueqzhY2A/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 tWZgR4foTlaIVwLXlFUvrA/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 ULbZobJ1SYC-ifMIExNx3w/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 ur44xWUXSvihHs_fNgq8iw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 3 00:00 VGpPCSC0QWSqNPJL_E8UiQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 VwQxLM0eS3adjJsXBVz-2A/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 W2XWUQwLTW-KSP4uz3Qclw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 xP7tAXkRRxi-_z_wtL1Nyw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 xric6DyaRBK7NT_YhN6gyA/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 3 00:00 y3TLsn6sS8m3-W45Zy1oJQ/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Feb 28 08:18 yFMw50bsS0CA-Be9Tp1flQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 yf__n5emTY-KbZBe3lJrCw/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 YNt1LbaVRMydpUmJ5czejQ/
drwxrwxrwx 3 elasticsearch elasticsearch 4096 Feb 28 07:30 Z8eIrCddQcaAxJuavm1iLQ/

Ben96 · March 5, 2018, 7:29am

Mhm, i don't think that's a Permission Issue because all your files/folders are 777 or 75X for somes.

Maybe folder where is mounted the var/lib/elasticsearch doesn't have the good System permission.

Is this "folder" mounted with option : rw ?

If not try to remount it : sudo mount -o remount,rw '/path/to/my_folder'

system · April 2, 2018, 7:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.