7.5.2 unable to revive connection (custom routine to generate certs)

Elastic Stack Kibana
1

Hi again,
Exists a way to see detailed kibana logs?
I have the famous "unable to revive connection" exception using a docker compose (se below)
When I pass verification mode to 'none' TLS works in cluster and kibana.
But when I switch to verification_mode: certificate issues begins.
I'm generating certs using cfssl gencert.
I'm using Centos 7
Yes I have CA and I can generate certs successfully (I used this to setup TLS in cluster)
Also I tried using LOGGING_VERBOSE=true but with no luck I don't have details about certs or authorities.
Thanks in advance for your help and comments.
Before post this thread I verified connection between containers and it's ok, additional as you can see I have a custom docker images, and I added to kibana the CA authorities to guarantee communications between containers, code I was added is:

RUN curl -o /etc/pki/ca-trust/source/anchors/cert.pem https://artifactory.abraxas.com/artifactory/sys-cacerts-dev/com/xoom/certificate-authority/cert.pem && \
    update-ca-trust force-enable && \
    update-ca-trust extract

And my abraxas.yml is:

version: '2.2'
services:
  elastic01:
    image: docker-registry.abraxas.com/infeng/elasticsearch:7.5.2-xpack
    container_name: elastic01
    hostname: elastic01
    environment:
        - node.name=elastic01
        - cluster.name=es-docker-cluster
        - discovery.seed_hosts=elastic02,elastic03
        - cluster.initial_master_nodes=elastic01,elastic02,elastic03
        - bootstrap.memory_lock=true
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        - xpack.license.self_generated.type=trial
        - xpack.security.enabled=true
        - xpack.security.http.ssl.enabled=true
        - xpack.security.http.ssl.key=/opt/abraxas/config/elasticsearch-7.5.2/certs/server-key.pem
        - xpack.security.http.ssl.certificate_authorities=/opt/abraxas/config/elasticsearch-7.5.2/certs/cert.pem
        - xpack.security.http.ssl.certificate=/opt/abraxas/config/elasticsearch-7.5.2/certs/server.pem
        - xpack.security.transport.ssl.enabled=true
        - xpack.security.transport.ssl.verification_mode=certificate
        - xpack.security.transport.ssl.certificate_authorities=/opt/abraxas/config/elasticsearch-7.5.2/certs/cert.pem
        - xpack.security.transport.ssl.certificate=/opt/abraxas/config/elasticsearch-7.5.2/certs/server.pem
        - xpack.security.transport.ssl.key=/opt/abraxas/config/elasticsearch-7.5.2/certs/server-key.pem
        - APPLY_LICENSE=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  elastic02:
    image: docker-registry.abraxas.com/infeng/elasticsearch:7.5.2-xpack
    container_name: elastic02
    hostname: elastic02
    environment:
        - node.name=elastic02
        - cluster.name=es-docker-cluster
        - discovery.seed_hosts=elastic01,elastic03
        - cluster.initial_master_nodes=elastic01,elastic02,elastic03
        - bootstrap.memory_lock=true
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        - xpack.license.self_generated.type=trial
        - xpack.security.enabled=true
        - xpack.security.http.ssl.enabled=true
        - xpack.security.http.ssl.key=/opt/abraxas/config/elasticsearch-7.5.2/certs/server-key.pem
        - xpack.security.http.ssl.certificate_authorities=/opt/abraxas/config/elasticsearch-7.5.2/certs/cert.pem
        - xpack.security.http.ssl.certificate=/opt/abraxas/config/elasticsearch-7.5.2/certs/server.pem
        - xpack.security.transport.ssl.enabled=true
        - xpack.security.transport.ssl.verification_mode=certificate
        - xpack.security.transport.ssl.certificate_authorities=/opt/abraxas/config/elasticsearch-7.5.2/certs/cert.pem
        - xpack.security.transport.ssl.certificate=/opt/abraxas/config/elasticsearch-7.5.2/certs/server.pem
        - xpack.security.transport.ssl.key=/opt/abraxas/config/elasticsearch-7.5.2/certs/server-key.pem
        - APPLY_LICENSE=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  elastic03:
    image: docker-registry.abraxas.com/infeng/elasticsearch:7.5.2-xpack
    container_name: elastic03
    hostname: elastic03
    environment:
        - node.name=elastic03
        - cluster.name=es-docker-cluster
        - discovery.seed_hosts=elastic01,elastic02
        - cluster.initial_master_nodes=elastic01,elastic02,elastic03
        - bootstrap.memory_lock=true
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        - xpack.license.self_generated.type=trial
        - xpack.security.enabled=true
        - xpack.security.http.ssl.enabled=true
        - xpack.security.http.ssl.key=/opt/abraxas/config/elasticsearch-7.5.2/certs/server-key.pem
        - xpack.security.http.ssl.certificate_authorities=/opt/abraxas/config/elasticsearch-7.5.2/certs/cert.pem
        - xpack.security.http.ssl.certificate=/opt/abraxas/config/elasticsearch-7.5.2/certs/server.pem
        - xpack.security.transport.ssl.enabled=true
        - xpack.security.transport.ssl.verification_mode=certificate
        - xpack.security.transport.ssl.certificate_authorities=/opt/abraxas/config/elasticsearch-7.5.2/certs/cert.pem
        - xpack.security.transport.ssl.certificate=/opt/abraxas/config/elasticsearch-7.5.2/certs/server.pem
        - xpack.security.transport.ssl.key=/opt/abraxas/config/elasticsearch-7.5.2/certs/server-key.pem
        - APPLY_LICENSE=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic
  kibana01:
    image: docker-registry.abraxas.com/infeng/kibana:7.5.2
    container_name: kibana01
    hostname: kibana01
    ports:
      - 5601:5601
    environment:
      - SERVER_NAME="localhost"
      - ELASTICSEARCH_URL="https://elastic01:9200"
      - ELASTICSEARCH_HOSTS=["https://elastic01:9200","https://elastic02:9200"]
      - ELASTICSEARCH_USERNAME=abraxasadmin
      - ELASTICSEARCH_PASSWORD=abraxas666
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=["/usr/share/kibana/config/certs/server.pem","/usr/share/kibana/config/certs/cert.pem"]
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY="/usr/share/kibana/config/certs/server-key.pem"
      - SERVER_SSL_CERTIFICATE="/usr/share/kibana/config/certs/server.pem"
      - TLS_ENABLED=true
      - LOGGING_VERBOSE=true
    networks:
      - elastic
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
networks:
  elastic:
    driver: bridge
2

Exists a way to see detailed kibana logs?

You can run Kibana with --verbose flag, it should output more information in console.

3

Hey @elemus,

I noticed you set the verification mode for your Elasticsearch nodes to certificate. Can you try the same for Kibana, and see if that helps?

ELASTICSEARCH_SSL_VERIFICATIONMODE=certificate

4

Thanks, trying...

5

I will try this too, thanks.

6

I solved my issue!!
To everyone with conectivity issues:

  1. pass to kibana this environment variables:
    • LOGGING_VERBOSE=true
    • LOGGING_QUIET=false
  2. after it you can see details about kibana and connections, if you are using tls you can check your connection certs using:
    openssl s_client -connect https://elastic01:9200
    this verify certs and show whats going on.

Thanks everyone for suggestions.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.