Elasticsearch started and running, but only for a few seconds

Elastic Stack Elasticsearch
1

I started elasticsearch on ubuntu. When I checked right after started, the status showed as active(running), but after just a few seconds the second status check showed failed. I've tried a few times and it always happened the same...Any comments? Here're the 2 status check results:

root@ip-172-31-89-133:/etc/init.d# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2018-10-21 02:57:34 UTC; 14s ago
Docs: http://www.elastic.co
Main PID: 24962 (java)
Tasks: 22 (limit: 1152)
CGroup: /system.slice/elasticsearch.service
├─24962 /usr/bin/java -Xms256m -Xmx256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true
└─25031 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Oct 21 02:57:34 ip-172-31-89-133 systemd[1]: Started Elasticsearch.
Oct 21 02:57:34 ip-172-31-89-133 elasticsearch[24962]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

root@ip-172-31-89-133:/etc/init.d# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2018-10-21 02:57:53 UTC; 4s ago
Docs: http://www.elastic.co
Process: 24962 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 24962 (code=exited, status=1/FAILURE)

Oct 21 02:57:34 ip-172-31-89-133 systemd[1]: Started Elasticsearch.
Oct 21 02:57:34 ip-172-31-89-133 elasticsearch[24962]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Oct 21 02:57:53 ip-172-31-89-133 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Oct 21 02:57:53 ip-172-31-89-133 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
root@ip-172-31-89-133:/etc/init.d#

2

That seems to be a very small heap, which could very well be causing problems. What does the Elasticsearch logs say?

3

I've tried bigger heap, but they failed right away:

root@ip-172-31-89-133:/etc/init.d# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)

Active: failed (Result: exit-code) since Sun 2018-10-21 03:20:35 UTC; 2s ago

Docs: http://www.elastic.co

Process: 25159 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)

Main PID: 25159 (code=exited, status=1/FAILURE)

Oct 21 03:20:35 ip-172-31-89-133 systemd[1]: Started Elasticsearch.

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Not enough space' (errno=12)

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: #

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: # There is insufficient memory for the Java Runtime Environment to continue.

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: # Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: # An error report file with more information is saved as:

Oct 21 03:20:35 ip-172-31-89-133 elasticsearch[25159]: # /var/log/elasticsearch/hs_err_pid25159.log

Oct 21 03:20:35 ip-172-31-89-133 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE

Oct 21 03:20:35 ip-172-31-89-133 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

root@ip-172-31-89-133:/etc/init.d#

4

with 1g heap, elasticservice failed to start, and the begining of the log shows:

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.

Possible reasons:

The system is out of physical RAM or swap space

The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap

Possible solutions:

Reduce memory load on the system

Increase physical memory or swap space

Check if swap backing store is full

Decrease Java heap size (-Xmx/-Xms)

Decrease number of Java threads

Decrease Java thread stack sizes (-Xss)

Set larger code cache with -XX:ReservedCodeCacheSize=

This output file may be truncated or incomplete.

Out of Memory Error (os_linux.cpp:2706), pid=25159, tid=25211

JRE version: (10.0.2+13) (build )

Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.2, mixed mode, aot, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)

Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /usr/share/elasticsearch/core.25159)

5

What is the specification of the host? Have you located and looked at the logs Elasticsearch generates?

6

I'm using EC2 t2.micro free, with 1CPU 8GB RAM...
instead of Elasticsearch.log, I found only some log files like /var/log/elasticsearch/hs_err_pid25159.log, so I just check the one with latest timestamp. That's what I showed in the previous post (only the beginning part, as the whole log is quite lengthy).

7

Hi folks,

I am trying to run Elasticsearch inside a Docker container on my Ubuntu laptop. And I experience quasi the same issue.

I dedicated 2GB of RAM to Elasticsearch container. The only message I see through docker logs command is the following:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

The exit code of the container, reported by Docker Swarm, is the 137 exit code.

For completeness, the docker-compose.yml

indent version: '3.3'
volumes:
   samplevolume:

networks:
  backend:
    driver: overlay
    driver_opts:
       encrypted: 'yes'

services:
  elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:6.5.0
     container_name: elasticsearch
     deploy:
       replicas: 1
       update_config:
       parallelism: 1
       delay: 10s
       restart_policy:
         condition: on-failure
       resources:
         limits:
          cpus: '1'
          memory: 2G
         reservations:
           cpus: '1'
           memory: 2G
   environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - "cluster.name=test-cluster"
      - "bootstrap.memory_lock=true"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
ulimits:
  memlock:
    soft: -1
    hard: -1
networks:
  - backend
volumes:
  - samplevolume:/usr/data/elasticsearch

kibana:
image: docker.elastic.co/kibana/kibana:6.5.0
ports: 
  - '5601:5601'
networks:
  - backend
environment:
  ELASTICSEARCH_URL: http://elasticsearch:9200

The same configuration worked with the version 6.2.3; now not any more.

Any help is very appreciated :slight_smile:

Thanks in advance!

UPDATE: Sorry for the bad indentation of the docker-compose file!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.