__es__.tmp

Elastic Stack Elasticsearch
1

Quite often I notice 'Service Unavailable' for Elasticsearch when accessing Kibana. I SSH into all of the nodes and start investigating logs.

Most frequently, it is a result of this file "es.tmp"

The error in the logs is as follows ::

Exception in thread "main" java.nio.file.FileAlreadyExistsException: /data/elasticsearch_data/TrainingProd/nodes/0/es.tmp
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at org.elasticsearch.env.NodeEnvironment.ensureAtomicMoveSupported(NodeEnvironment.java:794)
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:82)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at <<>>
at org.elasticsearch.node.Node.(Node.java:200)
at org.elasticsearch.node.Node.(Node.java:128)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

To fix this, I have to rename that file to something else, such as adding .old to the end. Then I restart Elasticsearch and the stack is fine again and accessible.

Does anyone know anything about this?? Is this a known ES bug?

Version of ES :: 2.2.1

2

Hi @Michael1,

I dug around in the code and I think this is a bug. I am just not sure were the problem occurs exactly.

If this problem happens again, can you please check whether there is also a file called __es__.final? Can you also check whether there is any related exception trace before this one (e.g. that a file could not be deleted)?

It would be great if you could look up this information and open a ticket on Github then.

Daniel

3

I can check the next time it happens. Is the information I provided above enough to open an issue on Git?

4

Hi,

it would help greatly if we know __es__.final also exists and also if there are any exception traces before the one you have provided (it is possible that either file deletion failed before or the problem occured when __es__.final was first referenced).

Daniel

5

@Michael1: Somebody opened a similar ticket today on Github: https://github.com/elastic/elasticsearch/issues/19036

6

Do you know if this continue happens with elasticsearch 2.4.1. Because I have the same problem with elasticsearch 2.4.1:
bash-4.1# service elasticsearch start
Starting elasticsearch: Exception in thread "main" java.nio.file.FileAlreadyExistsException: /data/elasticsearch/marta/nodes/0/es.tmp
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at org.elasticsearch.env.NodeEnvironment.ensureAtomicMoveSupported(NodeEnvironment.java:794)
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:82)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)