# Logstash pipeline processing

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-processing/298921>\
**Category:** Logstash\
**Created:** [March 6, 2022, 7:08pm UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921 "2022-03-06T19:08:12Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![KarlWolf](https://avatars.discourse-cdn.com/v4/letter/k/a6a055/32.png) [@KarlWolf](https://discuss.elastic.co/u/KarlWolf)\
**Post date:** [March 6, 2022, 7:08pm UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/1 "2022-03-06T19:08:12Z")

</div>

Hello,  
I am having a problem with Logstash configuration. I can't get rid with pipelines configuraton. My scenario:

logstash.yml - only lines which are uncommented/changed

```auto
path.data: /usr/share/logstash
pipeline.workers: 4
pipeline.batch.size: 1536
log.level: error
path.logs: /var/log/logstash

```

pipelines configuration:

```auto
- pipeline.id: files_log
  path.config: "/etc/logstash/pipelines/gzipped_data/01-checked_files.conf"
  pipeline.workers: 1
- pipeline.id: access_log
  path.config: "/etc/logstash/pipelines/access/02-access_log.conf"
  pipeline.workers: 2

```

First I thought that the problem occured on pipe configuration, but now it looks like problem with workers: my logstash has 4 workers: when logstash starts took one worker for files\_log and two workers for access\_log. Somehow (I thought workers stay always pinned to pipelines) when new file in files\_log appears logstash can't assign worker to pipeline and I must restart logstash process to workers re-assign.

Where I made a mistake on config? I can't find the error alone :\

Tanks & kind regards,  
Karl

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 6, 2022, 7:36pm UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/2 "2022-03-06T19:36:03Z")

</div>

> [@KarlWolf](#):
>
> `pipeline.workers: 4`

If you set that in logstash.yml then any pipelines that do not have a pipeline-specific setting for pipeline.workers will have 4 worker threads. Since both your pipelines _do_ have a pipeline-specific setting it really has no effect.

What does 01-checked\_files.conf look like?

---

<div class="post-metadata">

**Author:** ![KarlWolf](https://avatars.discourse-cdn.com/v4/letter/k/a6a055/32.png) [@KarlWolf](https://discuss.elastic.co/u/KarlWolf)\
**Post date:** [March 7, 2022, 7:28am UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/3 "2022-03-07T07:28:07Z")

</div>

Hello,  
Thanks for replay.  
Here is my configuration (input section with details, filter ans output are probably without issues (I didn't notice any problems with filter or saving logs in ES):

```auto
input {
  file {
    path => "/data/files-log/*.gz"
    sincedb_path => "/data/files-log.db"
    sincedb_clean_after => 5
    start_position => "beginning"
    mode => "read"
    file_completed_log_path => "/var/log/logstash/logstash-files-log_done.log"
    file_completed_action => "log_and_delete"
    exit_after_read => "true"
    type => "fileslog"
  }
}

filter {
  if [type]== "fileslog" {
    grok { ...}
    mutate { ... }
  }
}

output {
# stdout { codec => rubydebug }
  if [type] == "fileslog" {
    elasticsearch { ... }
  }
}

```

Thank you  
Kind regards,  
Karl Wolf

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 7, 2022, 11:54am UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/4 "2022-03-07T11:54:07Z")

</div>

> [@KarlWolf](#):
>
> `exit_after_read => "true"`

I suspect that you are not understanding what this does. The documentation is [here](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-exit_after_read).

---

<div class="post-metadata">

**Author:** ![KarlWolf](https://avatars.discourse-cdn.com/v4/letter/k/a6a055/32.png) [@KarlWolf](https://discuss.elastic.co/u/KarlWolf)\
**Post date:** [March 7, 2022, 9:33pm UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/5 "2022-03-07T21:33:47Z")

</div>

Hello Badger,  
Thanks for your answer. I deleted  
`exit_after_read => "true"`  
and looks better. .. but is not working fully right.  
Right now after restart: logstash proceeded over files which was on the folder and waited for the new one.  
After new file transfer Logstash processed three of six files (leave three without parsing). After next transfer (5 files) took again three of them. So it is better but is not ideal.

I don't get it why is not determistic...  
Regards,  
Karl Wolf

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 7, 2022, 9:52pm UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/6 "2022-03-07T21:52:43Z")

</div>

That sounds a lot like inode re-use. Read [this](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719/2) thread.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 4, 2022, 9:53pm UTC](https://discuss.elastic.co/t/logstash-pipeline-processing/298921/7 "2022-04-04T21:53:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.