Updated docker application logs not getting processed

Elastic Stack Logstash
1

Hi, I'm fairly new to Elastic Search, and I'm experiencing strange behaviour when re-deploying an application that sends its logs to ELK. The application is a Java app and using Filebeat to send the logs (both running in docker containers) to logstash, elastic search and kibana on a separate server, again all running in docker containers.

If I start with a clean setup, everything work as I would expect it to. If I then make changes to the java application, stop and restart the docker containers (including filebeat), then I can see no further logs in the ELK stack. Note, if I make no changes to the app and simply stop and restart the containers, everything continues to work.

If I now delete the image in the index management part of the web portal and restart everything, it starts to see the new application, but obviously I've now lost all existing data.

Is this expected behaviour? Is there any way I can fix this issue?

I have tried checking the logs in filebeat, logstash and elastic search, switching all to debug mode. I'm seeing lots of logging, but no relevant errors being reported by any of the applications.

If anyone knows what I'm doing wrong, please let me know.

2

I think I may have resolved the issue myself. It looks like it was related to Filebeat and the order in which the containers started.

If I was redeploying the application, and Filebeat started before the new application had started, it wasn't finding the log files. I've fixed it by using the following docker compose file (tested several times, so seems to work):

version: "3.5"
services:
  my-app:
    image: my-app:0.0.1
    ports:
      - "8084:8084"
    environment:
      - "SPRING_PROFILES_ACTIVE=docker"
    labels:
      collect_logs_with_filebeat: "true"
      decode_log_event_to_json_object: "true"

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.10.0
    command: filebeat -e -strict.perms=false                               # Needed for file permissions on config file
    volumes:
      - ./filebeat/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:rw # Configuration file
      - /var/lib/docker/containers:/var/lib/docker/containers:ro           # Docker logs
      - /var/run/docker.sock:/var/run/docker.sock:ro                       # Additional information about containers
      - ./filebeat/data:/usr/share/filebeat/data:rw                        # Persistence data
    user: root                                                             # Allow access to log files and docker.sock
    restart: on-failure
    depends_on:
      - my-app
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.