First time trying to run elk with docker-compose

Elastic Stack Kibana
1

Hi guys, im very interested setting up a centralized log server using docker / docker-compose:
After following oficial , and no oficial documentation:
https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html
https://github.com/deviantony/docker-elk

Got same result.
After running:
docker-compose up
Lines begin to run on the screen , but im no sure when stack is ready to use:
After one hour I connect to server on a second console and can see:

CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                                                                              NAMES
5a2a7760cd8f        docker-elk_kibana          "/usr/local/bin/dumb…"   9 minutes ago       Up 9 minutes        0.0.0.0:5601->5601/tcp                                                             docker-elk_kibana_1
352e379055d9        docker-elk_logstash        "/usr/local/bin/dock…"   9 minutes ago       Up 9 minutes        0.0.0.0:5000->5000/tcp, 0.0.0.0:9600->9600/tcp, 0.0.0.0:5000->5000/udp, 5044/tcp   docker-elk_logstash_1
2c27c53d6893        docker-elk_elasticsearch   "/tini -- /usr/local…"   9 minutes ago       Up 9 minutes        0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                     docker-elk_elasticsearch_1'

Everything seems ok from there, but when try connecting from browse , I got:
Kibana server is not ready yet

I have no idea where to beging debugin this ... some questions came to my mind:
How long does the stack to be ready ? How should I check ?
Where should I look for errors ?

Also read about assign 4Gb to the stack.
This is what I did:
sysctl -w vm.max_map_count=262144
Im trying to read about this, so any advice would be wellcome.

btw:
I did not change any config file at all.
Im running statck on fresh centos 7 vm with 8Gb memory.

2

Hi @leostereo

It should only take a couple minutes to come up

Can you post your docker compose file?

Did you look at the log and see and errors? If you run docker compose from the command line the logs get directed to stdout / console. Look for errors.

If you are running on a machine with 8GB of RAM you should be OK assuming you there are not other processes that have already consumed the memory or you did not increase the Elasticsearch heap setting up too high.

Did you try the following commands to see in elasticsearch came up?

curl -X GET "localhost:9200
curl -X GET "localhost:9200/_cat/nodes?v&pretty

You can try this compose file I just ran it ... it works. It is a sample it is not made for production but its a simple test.

You run it with the following command.

TAG=7.9.0 docker-compose

---
  version: '3.1'
  services:
    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
      environment: ['ES_JAVA_OPTS=-Xms2g -Xmx2g','bootstrap.memory_lock=true','discovery.type=single-node', 'http.host=0.0.0.0', 'transport.host=127.0.0.1']
      ports: ['127.0.0.1:9200:9200']
      networks: ['stack']
      ulimits:
        memlock:
          soft: -1
          hard: -1
        nofile:
          soft: 65536
          hard: 65536
  
    kibana:
      image: docker.elastic.co/kibana/kibana:${TAG}
      ports: ['127.0.0.1:5601:5601']
      networks: ['stack']
      depends_on: ['elasticsearch']
  
  networks: {stack: {}}
3

Dear @stephenb
Thanks for your words.
The yml file you share works ok on my local pc but not on my remote vm.

The file im using is: (I did not change nothing here, just copy/paste it)

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    ports:
      - 9201:9201
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    ports:
      - 9202:9202
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.9.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

Also , the output you mention shows an error:

[root@deploy ~]#  curl -X GET "localhost:9200"
{
  "name" : "es01",
  "cluster_name" : "es-docker-cluster",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.9.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "a479a2a7fce0389512d6a9361301708b92dff667",
    "build_date" : "2020-08-11T21:36:48.204330Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@deploy ~]#  curl -X GET "localhost:9200/_cat/nodes?v&pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

About log error ... this is what I can show (trunked):

[root@deploy ~]# docker logs 187 | grep -E "error|invalid"
[BABEL] Note: The code generator has deoptimised the styling of /usr/share/kibana/x-pack/plugins/canvas/server/templates/pitch_presentation.js as it exceeds the max of 500KB.
{"type":"log","@timestamp":"2020-08-26T23:29:32Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-08-26T23:29:32Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-08-26T23:29:34Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET http://es01:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect EHOSTUNREACH 172.26.0.4:9200"}
{"type":"log","@timestamp":"2020-08-26T23:29:34Z","tags":["error","elasticsearch","monitoring"],"pid":6,"message":"Request error, retrying\nGET http://es01:9200/_xpack => connect EHOSTUNREACH 172.26.0.4:9200"}
{"type":"log","@timestamp":"2020-08-26T23:29:34Z","tags":["error","elasticsearch","monitoring"],"pid":6,"message":"Request error, retrying\nGET http://es01:9200/_xpack => connect EHOSTUNREACH 172.26.0.4:9200"}
{"type":"log","@timestamp":"2020-08-26T23:29:35Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-26T23:29:37Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-08-26T23:29:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-26T23:30:04Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-26T23:30:34Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-26T23:31:04Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-26T23:31:34Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-26T23:32:04Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

mmmm .... thanks again.
im lost here.
LEo

4

Hi @leostereo

Closer look at :
https://www.elastic.co/guide/en/elasticsearch/reference/7.9/docker.html#_set_vm_max_map_count_to_at_least_262144

I see you ran
sysctl -w vm.max_map_count=262144

I believe what you really need to do is the following

The vm.max_map_count setting should be set permanently in /etc/sysctl.conf which means you need to edit the /etc/sysctl.conf file and append vm.max_map_count=262144 to the bottom otherwise it is not taking affect.

I would clean up everything after you set that and try again

That is what I did to get your compose file working.

I waited a couple minutes then ran the following and everything looks good!

$ curl localhost:9200/_cat/nodes?v
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.32.5           34          92   9    0.33    0.63     1.27 dilmrt    -      es02
192.168.32.2           51          92   9    0.33    0.63     1.27 dilmrt    -      es03
192.168.32.3           37          92   9    0.33    0.63     1.27 dilmrt    *      es01
5

Thanks @stephenb
You were right, about sysctrl.conf.
Now error message is different but it seems closer to work.

[root@deploy ~]# curl localhost:9200/_cat/nodes?v
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Im reading about this error message.
Seems to be an internal comunication problem between stack elements.
Any update I will let you know.
Leandro.

6

dear @stephenb, thanks for your words.
Final idea of elk , is to collect netflow so ... I change to regular installation.
ELK stack is working now.
Thanks.

1 Like
7

@leostereo
Good to hear.
Did you ever get the docker working?
That said ELK direct on host should be higher performing.
So you have a 3 node cluster working with direct installation?

Make sure you checkout the Filebeat Netflow Module
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-netflow.html

Good Luck!

8

What do you mean with 3 nodes?
This I what i can see:

[root@deploy ~]# curl -X GET "localhost:9200/_cat/nodes?v&pretty"
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           54          79  99    7.38    7.06     7.14 dilmrt    *      deploy

How can I check the overall status of the stack?

9

That is 1 node, that will be fine for POC. Best practice a production cluster will have more than 1 node for performance and HA and scale.

curl -X GET "localhost:9200/"
curl -X GET "localhost:9200/_cat/health"

Health will most likely show Yellow because there is no replication (no copies of the data so you are at risk of losing data if the node dies)

Next you should log into Kibana and poke around.

At this point I would recommend some of the free training and webinars and blogs, you have much to learn.
https://www.elastic.co/videos/
https://www.elastic.co/training/free
https://www.elastic.co/blog/

And if you are going to be devops for a Production Elastic I would get some actual real training
https://www.elastic.co/training/

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.