ElasticSearch stop on Centos 7

[skazhikadyadya]

Hi. I have installed ES 5.6.16 on my Centos 7.3 from yum.

After installation I was check status with service elasticsearch status:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-07-11 04:20:07 CEST; 8min ago
Docs: http://www.elastic.co
Process: 30621 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p {PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs={LOG_DIR} -Edefault.path.data={DATA_DIR} -Edefault.path.conf={CONF_DIR} (code=exited, status=1/FAILURE)
Process: 30618 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 30621 (code=exited, status=1/FAILURE)

Ok. In /var/log/elasticsearch/elasticsearch.log I see:

1) Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:12, legacy:false, file:/var/lib/elasticsearch/nodes/0/indices/_L3XAhS3TFu3I8r0lyXm0Q/_state/state-12.st]]; nested: IOException[failed to read [id:12, legacy:false, file:/var/lib/elasticsearch/nodes/0/indices/_L3XAhS3TFu3I8r0lyXm0Q/_state/state-12.st]]; nested: IllegalArgumentException[Unexpected field [mapping_version]];
  at org.elasticsearch.gateway.GatewayMetaState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayMetaState
    for parameter 4 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayService
Caused by: ElasticsearchException[java.io.IOException: failed to read [id:12, legacy:false, file:/var/lib/elasticsearch/nodes/0/indices/_L3XAhS3TFu3I8r0lyXm0Q/_state/state-12.st]]; nested: IOException[failed to read [id:12, legacy:false, file:/var/lib/elasticsearch/nodes/0/indices/_L3XAhS3TFu3I8r0lyXm0Q/_state/state-12.st]]; nested: IllegalArgumentException[Unexpected field [mapping_version]];
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:196)
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:335)
        at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:90)
        at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:128)
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:91)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)

How I can fix it?

Thanks!

[rugenl (Len Rugen)]

Has any other version of elastic ever been installed on this server?

Are there files in the data directory? Is the owner and permissions correct? If this is a new stack, you could just delete everything in the data directory and start over.

I think I've seen this in a lab where Elastic was started from the command line (as root), created files in the data directory owned by root, so it would fail started as elastic by systemctl.

[skazhikadyadya]

Thank you for reply.

Has any other version of elastic ever been installed on this server?
No.

Are there files in the data directory? Is the owner and permissions correct? If this is a new stack, you could just delete everything in the data directory and start over.
Where I can find that? Is it normal?
[root@shop ~]# ls -al /var/lib/elasticsearch/
total 12
drwxrwx--- 3 elasticsearch elasticsearch 4096 Mar 13 16:34 .
drwxr-xr-x. 37 root root 4096 Jul 2 23:23 ..
drwxrwsr-x 3 elasticsearch elasticsearch 4096 Jul 2 23:23 nodes

I think I've seen this in a lab where Elastic was started from the command line (as root), created files in the data directory owned by root, so it would fail started as elastic by systemctl.
I can't install ES without root, am i wrong?

[skazhikadyadya]

I deleted the data in /var/lib/elasticsearch and started ES again. It works! Thank you!