# Separating pipeline logs issues

**URL:** <https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [August 6, 2023, 12:19pm UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199 "2023-08-06T12:19:46Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [August 6, 2023, 12:19pm UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/1 "2023-08-06T12:19:46Z")

</div>

Hi guys.  
I am running Logstash version 8.8.2 on docker with more than 10 pipelines.  
I want to separate pipeline logs to separate log files, according to logstash document:

> **[logstash.yml | Logstash Reference \[8.8\] | Elastic](https://www.elastic.co/guide/en/logstash/8.8/logstash-settings-file.html)**

document said set `path.logs` and `pipeline.separate_logs` in `logstash.yml` to separate pipeline logs.  
I set those parameters but not working.  
are there any other options to set?  
**Note** : I want to separate the logs that the container shows not the pipeline itself output or result.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 6, 2023, 5:49pm UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/2 "2023-08-06T17:49:49Z")

</div>

Can you share both your `pipelines.yml` and `logstash.yml` files?

---

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [August 6, 2023, 6:05pm UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/3 "2023-08-06T18:05:11Z")

</div>

yes sure,

logstash.yml

```auto
http.host: "0.0.0.0"
node.name: "logstash-coz-ls-transaction"
xpack.monitoring.elasticsearch.hosts: ["${ELASTIC_HOST}"]
monitoring.enabled: false
monitoring.cluster_uuid: "${ELASTIC_CLUSTER_UUID}"
## X-Pack security credentials
#
xpack.monitoring.elasticsearch.username: ${ELASTIC_ROOT_USER}
xpack.monitoring.elasticsearch.password: "${ELASTIC_ROOT_PASS}"
api.auth.type: basic
api.auth.basic.username: ${ELASTIC_ROOT_USER}
api.auth.basic.password: "${ELASTIC_ROOT_PASS}"
allow_superuser: true
config.reload.automatic: true
  # This value set because of this warning: The default value of `api.auth.basic.password_policy.mode` will change to `ERROR` in a future release of Logstash. Set it to `ERROR` to observe the future behavior early, or set it to `WARN` to lock in the current behavior.
api.auth.basic.password_policy.mode: ERROR
path.logs: "/usr/share/logstash/logs"
pipeline.separate_logs: true
log.level: info

```

pipelines.yml:

```auto
- pipeline.id: ${PIPEID_COZ_PURCHASE_NOZOMI}
  path.config: "./pipeline/${PIPEID_COZ_PURCHASE_NOZOMI}/*.conf"

- pipeline.id: ${PIPEID_COZ_PURCHASE_DSELL}
  path.config: "./pipeline/${PIPEID_COZ_PURCHASE_DSELL}/*.conf"

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 6, 2023, 6:44pm UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/4 "2023-08-06T18:44:58Z")

</div>

> [@Siavash\_Fazli](#):
>
> I set those parameters but not working.

So, can you provide some context of what is not working? Like, share some of the logs you are getting.

You said that you have more than 10 pipelines, but in your `pipelines.yml` you have just 2 pipelines, so you should get only 2 log files, one for each pipeline id after changing the `pipeline.separate_logs` setting and restarted logstash.

---

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [August 8, 2023, 11:59am UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/5 "2023-08-08T11:59:04Z")

</div>

everything works fine, pipelines too. but Log files have not been created.  
I don't have any problem with reading logs or errors. this is the output of `docker compose logs -f` :

```auto
logstash-competition | [2023-07-03T13:49:31,280][INFO][logstash.inputs.jdbc][coz-ls-pipe-battlelog-special][ca1ad565781114828ae3376f73c8954e387ca45e3bad51ab2dd177af988256bb] (0.690862s) SELECT count(*) AS `count` FROM (SELECT nozomi_clan_hero.id AS user_id,
logstash-competition | nozomi_clan_hero.hid AS attack_hero_id,
logstash-competition | nozomi_clan_hero.ctime AS attack_time,
logstash-competition | nozomi_clan_hero.btime AS attack_purchase_time,
logstash-competition | nozomi_clan_hero.htime AS attack_reward_collection_time,
logstash-competition | nozomi_clan_hero.anum AS attack_count,
logstash-competition | nozomi_clan_hero.buys AS attack_count_purchase,
logstash-competition | nozomi_clan_hero.hrwd AS attack_reward,
logstash-competition | nozomi_clan_hero.anum - nozomi_clan_hero.buys AS attack_count_free,
logstash-competition | crystal_table.value AS user_crystal,
logstash-competition | chip_table.value AS user_chip
logstash-competition | 	
logstash-competition | FROM nozomi_clan_hero
logstash-competition | LEFT JOIN user_properties AS crystal_table 
logstash-competition | ON ( nozomi_clan_hero.id = crystal_table.id AND crystal_table.pid = 4 ) 
logstash-competition | 
logstash-competition | LEFT JOIN user_properties AS chip_table
logstash-competition | ON ( nozomi_clan_hero.id = chip_table.id AND chip_table.pid = 7 )
logstash-competition | 
logstash-competition | WHERE nozomi_clan_hero.ctime > 1688392144 AND nozomi_clan_hero.ctime <= UNIX_TIMESTAMP(NOW())
logstash-competition | 
logstash-competition | ORDER BY nozomi_clan_hero.ctime ASC) AS `t1` LIMIT 1

```

* * *

yes, I have more than 10 pipelines but on other containers.  
Since I separated containers, debugging got easier. but I couldn't separate logs yet.  
my problem is log files didn't create.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 8, 2023, 1:16pm UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/6 "2023-08-08T13:16:54Z")

</div>

> [@Siavash\_Fazli](#):
>
> Since I separated containers, debugging got easier. but I couldn't separate logs yet.  
> my problem is log files didn't create.

I do not use Docker, but looking how the container is built in the [elastic github](https://github.com/elastic/dockerfiles/tree/8.9/logstash), it seems that it per default will log to stdout, no matter if you have set `path.logs` or not, this is mentioned in the [documentation](https://www.elastic.co/guide/en/logstash/current/docker-config.html#_logging_configuration).

Looking at the [docker-entrypoint](https://github.com/elastic/dockerfiles/blob/8.9/logstash/bin/docker-entrypoint) file, it seems to use an environment variable to change this behavior, setting `$LOG_STYLE` as `file` would tell it to log to files, but this is not mentioned in the documentation.

I think that if you provide this environment variable with the value of `file` it will log to files, but I'm not sure since I do not use docker.

You may try it or wait for someone from Elastic to provide more information.

---

<div class="post-metadata">

**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\
**Post date:** [September 4, 2023, 8:46am UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/7 "2023-09-04T08:46:00Z")

</div>

Thank you. I'll check it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 2, 2023, 8:46am UTC](https://discuss.elastic.co/t/separating-pipeline-logs-issues/340199/8 "2023-10-02T08:46:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.