The Elasticsearch class is not starting on an exisiting project, but working fine on a vanilla site?

[truls1502 (Truls)]

I am trying to use DDEV, a Local Development environment, and use elasticsearch 6.8 with Drupal 8 site.

I started to use the example recipe. But since it is not working and missing some parameters, so I want to get it working and contribute back ddev_contrib with a elastic-stack (elasticsearch and kibana). But I have used almost two whole days with no luck to find the reason.

On a vanilla Drupal site, it worked with my setup. But not on an existing project which I am getting elasticsearch exited or died.

My docker-compose-elastic-stack.yaml looks:

version: '3.6'
services:
  elasticsearch:
    container_name: ddev-${DDEV_SITENAME}-elasticsearch
    hostname: ${DDEV_SITENAME}-elasticsearch
    image: elasticsearch:6.8.0
    ports:
      - "9200"
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - VIRTUAL_HOST=$DDEV_HOSTNAME
      - HTTP_EXPOSE=9200
    ulimits:
      memlock:
        soft: -1
        hard: -1
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
      - ".:/mnt/ddev_config"

  kibana:
    container_name: ddev-${DDEV_SITENAME}-kibana
    hostname: ${DDEV_SITENAME}-kibana
    image: kibana:6.8.0
    depends_on:
      - elasticsearch
    ports:
      - "5601"
    environment:
      - VIRTUAL_HOST=$DDEV_HOSTNAME
      - HTTP_EXPOSE=5601
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    volumes:
      - ".:/mnt/ddev_config"

  web:
    links:
      - elasticsearch:elasticsearch

volumes:
  elasticsearch:
    name: "${DDEV_SITENAME}-elasticsearch"

[truls1502 (Truls)]

When I do ddev log -s elasticsearch the log says on an exisiting project:

ddev logs -s elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-11-22T12:49:52,563][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:218) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:270) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
	at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
	at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
	at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
	at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:270) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 6 more
[...... etc]

[truls1502 (Truls)]

The problem was resolved. I had to remove the old volume which I might have created before by mistake configurations, then rebuild it again and now it works as I wanted in both vanilla site and existing project. Thanks to Randy Fay from DDEV.

To check the old volume, I used docker volume ls | grep elastic then delete the unwanted volume or the volume which I had problem with.