# Logstash log separation per pipeline doesn't work!

**URL:** <https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [February 8, 2023, 6:45am UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975 "2023-02-08T06:45:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [February 8, 2023, 6:45am UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/1 "2023-02-08T06:45:11Z")

</div>

Hi guys.

We use ELK version 8.4.2.  
on logstash, we have 15 pipelines. Now we need to separate the logs.  
There is a solution in the elastic document:

> **[logstash.yml | Logstash Reference \[8.4\] | Elastic](https://www.elastic.co/guide/en/logstash/8.4/logstash-settings-file.html)**

This document says to insert 2 directives in **logstash.yml** that automatically separate logs in separate files per pipeline:

```auto
path.logs: LOGSTASH_HOME/logs

pipeline.separate_logs: true

```

and this is my **logstash.yml**

```auto
## Default Logstash configuration from Logstash base image.
## https://github.com/elastic/logstash/blob/master/docker/data/logstash/config/logstash-full.yml
#
http.host: "0.0.0.0"
node.name: "coz-logstash"
xpack.monitoring.elasticsearch.hosts: ["${ELASTIC_HOST}"]

## X-Pack security credentials
#
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: ${ELASTIC_ROOT_USER}
xpack.monitoring.elasticsearch.password: ${ELASTIC_ROOT_PASS}
api.auth.type: basic
api.auth.basic.username: ${ELASTIC_ROOT_USER}
api.auth.basic.password: ${ELASTIC_ROOT_USER}

config.reload.automatic: true
pipeline.separate_logs: true
path.logs: "/usr/share/logstash/log/"

```

these settings don't write any log file in the **path.logs** directory.  
Are there any other settings that I have to set?

I change permissions to 777 (for testing) but still, there isn't any log file.

can anyone help me?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 8, 2023, 5:34pm UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/2 "2023-02-08T17:34:11Z")

</div>

Have you set `pipeline.id`?

> If enabled Logstash will create a different log file for each pipeline, using the pipeline.id as name of the file.

---

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [February 12, 2023, 6:47am UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/3 "2023-02-12T06:47:51Z")

</div>

Hi @Rios.  
yes I set the `pipeline.id` .  
After 3, or 4 days doesn't exist any log file.

This is my `pipelines.yml` :

```auto
- pipeline.id: ${PIPEID_COZ_PURCHASE_NOZOMI}
  path.config: "./pipeline/${PIPEID_COZ_PURCHASE_NOZOMI}/*.conf"
  pipeline.workers: 2

- pipeline.id: ${PIPEID_COZ_PURCHASE_DSELL}
  path.config: "./pipeline/${PIPEID_COZ_PURCHASE_DSELL}/*.conf"
  pipeline.workers: 2

- pipeline.id: ${PIPEID_COZ_BATTLE_SERVER}
  path.config: "./pipeline/${PIPEID_COZ_BATTLE_SERVER}/*.conf"

- pipeline.id: ${PIPEID_COZ_USER_STATIC}
  path.config: "./pipeline/${PIPEID_COZ_USER_STATIC}/*.conf"

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 12, 2023, 12:09pm UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/4 "2023-02-12T12:09:46Z")

</div>

The pipeline log naming is working perfectly on my side.

You should add $ at the begging `path.logs : ${LOGSTASH_HOME}/config`  
Also, it's useful to set temporarily to see LS settings values:  
`config.debug: true`  
`log.level: debug`

---

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [February 12, 2023, 12:22pm UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/5 "2023-02-12T12:22:32Z")

</div>

@Rios Where can I see the results of these setting?

```auto
config.debug: true
log.level: debug

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 12, 2023, 12:57pm UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/6 "2023-02-12T12:57:44Z")

</div>

The command/terminal line.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 12, 2023, 5:36pm UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/7 "2023-02-12T17:36:24Z")

</div>

Also, check that /etc/logstash/log4j.properties contains the routing appender

```auto
appender.routing.type = PipelineRouting
appender.routing.name = pipeline_routing_appender
appender.routing.pipeline.type = RollingFile
appender.routing.pipeline.name = appender-${ctx:pipeline.id}
appender.routing.pipeline.fileName = ${sys:ls.logs}/pipeline_${ctx:pipeline.id}.log
...

```

If you upgraded an old install to 8.4.2 it is possible that you have a log4j2.properties.rpmnew with that feature and a log4j2.properties without.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 12, 2023, 6:14pm UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/8 "2023-02-12T18:14:17Z")

</div>

Useful note, thanks Badger. I usually upgrade with new yml/properties, just append modified values from older configuration files.

---

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [February 28, 2023, 6:57am UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/9 "2023-02-28T06:57:58Z")

</div>

@Rios @Badger

I implemented options that you dear guys wrote, and checked several times but none of them doesn't work,

We run the logstash on docker and I found out we handle logging in docker-compose by these setting:

```auto
x-logging:
  &logging-default
  driver: "json-file"
  options:
    max-file: "5"
    max-size: "20m"

services:
  logstash:
    container_name: docker_logstash
    build:
      context: conf/coz/
      args:
        ELK_VERSION: '8.4.2'
    env_file:
      - ./conf/coz/logstash.env
    logging: *logging-default
    ports:
      - 9600:9600
      - 5044:5044

```

Could this setting have conflicted?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2023, 6:58am UTC](https://discuss.elastic.co/t/logstash-log-separation-per-pipeline-doesnt-work/324975/10 "2023-03-28T06:58:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.