Setting up ELK Stack for Kafka Logs Monitoring

Elastic Search
1

Hello all,

I'm a beginner, so please bear with anything I get wrong.

I have the task of setting up Logs Monitoring for Kafka with the ELK Stack. I didn't have any luck with versions 7.14 so I thought going to the newest version (8.15) would be my next best attempt.

I'm using docker compose in a Azure VM with Ubuntu to run Kafka broker, connect, controller, ELK Stack and Filebeat. For Kafka I'm using the "confluentinc/cp-server:7.6.0" image and have this:

"KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'"

in my environment variables for the Kafka logs.
I managed to get the ELK Stack running at least and can access it under "0.0.0.0:5601" .
I cannot however get Filebeat to start, as it gives the error:

error connecting to Elasticsearch at http://...:9200: Get "http://...:9200": EOF

from elasticsearch logs:

"WARN", "message":"received plaintext http traffic on an https channel, closing connection

Any advice on how to set this up further would be greatly appreciated :slight_smile:

Also with previous versions I was able to get some logs showing up in the Discover Tab but their "message" part was always full of weirdly encoded signs and not really comprehensible.

2

Hey @Elgis_Migle It looks like you setup ES cluster with security and SSL enabled, therefore it expects SSL traffic on https://...:9200 and not http://...:9200

You can see the warn log from ES that it's intentionally dropping the connection

"WARN", "message":"received plaintext http traffic on an https channel, closing connection
3

I was advised to switch to a more current version, so I did and now am on 8.15.

elasticsearch:
    image: elasticsearch:8.15.0
    container_name: elasticsearch
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - "/home/Admin/monitoring/elk_data:/usr/share/elasticsearch/data/"
    environment:
      - discovery.type=single-node
      - http.host=0.0.0.0
      - transport.host=0.0.0.0
      - cluster.name=elasticsearch
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=changeme
      - bootstrap.memory_lock=true
      - xpack.security.enrollment.enabled=true
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"

However when I now go to x.x.x.x:9200 I am prompted to enter a user and password. With elastic and changeme I cannot authenticate however:

"log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]"

and Kibana now logs:
[ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED x.y.z.w:9200

Elastic-agent also has problems:
"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: tls: first record does not look like a TLS handshake"

  elastic-agent:
      image: docker.elastic.co/beats/elastic-agent:8.15.0
      container_name: elastic-agent
      restart: always
      user: root
      environment:
        - FLEET_SERVER_ENABLE=true
        - FLEET_SERVER_ELASTICSEARCH_HOST=https://elasticsearch:9200
        - FLEET_ENROLLMENT_TOKEN=tokencopiedfromkibana
        - FLEET_SERVER_SERVICE_TOKEN=othertokencopiedfromkibana
        - FLEET_INSECURE=true
        - FLEET_URL=https://ipofmyazurevm:8220
        - FLEET_ENROLL=1

also what's the service token and enrollment token? In the page for managing the enrollment tokens it says not what token is for what.

4

@Elgis_Migle It’s a bit of a long shot, but I vaguely remember seeing a message like Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic] when I accidentally kept multiple ES containers (binding to same address) running on the same machine. Could you please check to ensure that all old ES containers (different from this 8.15 version) have been stopped?

For deploying latest ELK stack with Docker please refer to documentation Install Elasticsearch with Docker | Elasticsearch Guide [8.15] | Elastic, specifically look at docker-compose.yml (and .env) example. This should work well :slight_smile:

5

Yeah

ELASTICSEARCH_PASSWORD

Seems not the one I'm using nowadays but

ELASTIC_PASSWORD

HTH

6

I don't think there is another container running. I use a single docker-compose.yml and there is only one elasticsearch in it. "docker ps" also shows only one elasticsearch.

I adapted the docker-compose.yml you sent and Kibana and Elasticsearch work fine now, I think. Thanks!

elastic-agent however just logs this:
elastic-agent | {"log.level":"error","@timestamp":"2024-08-26T09:56:44.982Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: tls: failed to verify certificate: x509: certificate signed by unknown authority","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

With this setup:

elastic-agent:
      image: docker.elastic.co/beats/elastic-agent:8.15.0
      container_name: elastic-agent
      restart: always
      user: root
      environment:
        - FLEET_SERVER_ENABLE=true
        - FLEET_SERVER_ELASTICSEARCH_HOST=https://elasticsearch:9200
        - FLEET_ENROLLMENT_TOKEN=copied enrollment token
        - FLEET_SERVER_SERVICE_TOKEN=copied service token
        - FLEET_INSECURE=true
        - FLEET_URL=https://0.0.0.0:443
        - FLEET_ENROLL=1

In "Add a Fleet Server" under "Get started with Fleet Server" I put name "fleet" and URL "https://0.0.0.0" which gave me "https://0.0.0.0:443".
"Confirm connection" is however not showing any change and is just refreshing

@dadoonet Yeah, thanks for pointing that out.

7

@Elgis_Migle Since you’re running ES locally with SSL enabled, you’ll need to manage the SSL certificates as well. In the Docker Compose file I referenced, the certificate is generated and stored on a shared volume, allowing each container to access and use it for SSL. You can see how this is set up in the example docker-compose.yml for e.g. Kibana (you attach the certs volume and pass ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES). A similar setup will be needed for the fleet server too (to connect to ES). I’m not super familiar with its specific configuration, but the docs should help with that.

1 Like
8

Hello friend,

Maybe also keep in mind that log.dirs for kafka means the data directory, not the logging directory.

For kafka, the messages are called logs, which has partitions and offsets :slight_smile:

Regards.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.