How to add new config file to logstash which is created by docker compose?

Elastic Stack Logstash
1

How to add a new config file to logstash which is created by docker-compose?
I created my service with docker-compose and log stash is connected to kibana and elasticsearch.

Now I created a new config file for logstash but I don't know how to use it. I created the file in folder /elk/logstash/ in the host system. Unfortunately, all of the tutorials are based on the definition of the config file during the creation of the container, but I am seeking a way to update it after creation.
To make it more informative, I use ubuntu 22.04 in a VPS, and here is the docker-compose creation yml file:

version: '3.6'
services:
  Elasticsearch:
    image: elasticsearch:7.16.2
    container_name: elasticsearch
    restart: always
    volumes:
    - elastic_data:/usr/share/elasticsearch/data/
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      discovery.type: single-node
    ports:
    - '9200:9200'
    - '9300:9300'
    networks:
      - elk

  Logstash:
    image: logstash:7.16.2
    container_name: logstash
    restart: always
    volumes:
    - ./logstash/:/logstash_dir
    command: logstash -f /logstash_dir/logstash.conf
    depends_on:
      - Elasticsearch
    ports:
    - '9600:9600'
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk

  Kibana:
    image: kibana:7.16.2
    container_name: kibana
    restart: always
    ports:
    - '5601:5601'
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    depends_on:
      - Elasticsearch
    networks:
      - elk
volumes:
  elastic_data: {}

networks:
  elk:

I executed the docker up in the /elk/ folder and now I have the file in the pipeline folder but I don't know how to trigger logstash to pick it while it is created. Please make sure your answer is not based on the creation of the container again by "docker-compose up" which is not my point. Thanks

2

Please share your logstash and a sample pipeline configuration that does not work.

If you are dropping a pipeline in the pipeline folder, you may also need to include a reference to it in the pipelines.yml file Multiple Pipelines | Logstash Reference [8.12] | Elastic or reference it in your logstash.yml

3

The problem is that you are running the command line that explicitly only runs a specific logstash conf file.

As @strawgate said use pipeline.yml or simple load multiple conf files in the conf.d directory

1 Like
4

Hi!

The way I do it is through a volume. In the Logstash: service definition, under volumes:, I have one pointing straight to the pipeline:

- logstash-pipeline:/usr/share/logstash/pipeline

Then I simply drop my .yml files (yes, I have multiple different files for different tasks) in that directory, on the host. You could also copy your .yml file(s) to the pipeline, but will still need the volume for persistence:

docker cp <your.yml> logstash:/usr/share/logstash/pipeline/

Then restart the logstash container for the changes to take effect.

As pointed out by @stephenb, you should not use the command line in your docker-compose.yml. Logstash will load any .yml file it finds in its pipeline directory.

Hope this helps!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.