# Why does \`--path.settings\` delete events? Why does \`-f\` work perfectly?

**URL:** <https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592>\
**Category:** Logstash\
**Created:** [March 3, 2024, 5:06pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592 "2024-03-03T17:06:24Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [March 3, 2024, 5:06pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/1 "2024-03-03T17:06:24Z")

</div>

When I start logstash with `-f`, everything works perfectly. If I start logstash with `--path.settings`, logstash deletes events received from the input. Does anyone know why?

Here's how to reproduce the issue...i start with a success case followed by a failed case.

**Scenario 1: -f works perfectly**

I have an index called `my-alerts` with a simple mapping like this:

```auto
{
    "properties": {
      "alert_type": {
        "type": "keyword"
      },
      "hits": {
        "type": "keyword"
      }
    }
}

```

I created a file called `/etc/logstash/conf.d/my-alerts.conf` like this:

```auto
input {
  elasticsearch {
    schedule => "* * * * *"
    hosts => "${ES_HOST}"
    user => "${ES_USER}"
    password => "${ES_PASS}"
    index => "my-alerts"
    query => '{"query":{"bool":{"must":[{"match":{"alert_type":"changes"}}]}}}'
    ssl_enabled => true
    ssl_certificate_authorities => "${ES_CA_FILE}"
    ssl_verification_mode => "full"
  }
}
filter {
  json {
    source => "hits"
    target => "hitsjson"
  }
}
output {
  stdout {}
}

```

I created a `run.sh` like this:

```auto
ES_HOST="es01"
ES_USER="elastic"
ES_PASS="changeme"
ES_CA_FILE="/etc/certs/es/ca.crt"
export ES_HOST
export ES_USER
export ES_PASS
export ES_CA_FILE

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/my-alerts.conf --config.reload.automatic

```

When I run the command `~/run.sh` everything works perfectly.

**Scenario 2: --path.settings deletes the hits/hitsjson fields**

However, if I replace the line

```auto
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/my-alerts.conf --config.reload.automatic

```

with this line:

```auto
/usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic

```

then I will not see the events/fields `hits` or `hitsjson`. Why does using `/usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic` delete the `hits/hitsjson` fields?

* * *

Also this is my `/etc/logstash/pipelines.yml`

```auto
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 3, 2024, 5:14pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/2 "2024-03-03T17:14:28Z")

</div>

Most likely because then logstash is not using your conf file..

if you Do not use `-f` logstash will look for the pipelines.yml file for which pipeline to run

Check to see if you have a pipelines.yml if you want logstash to autoload conf.d remove that file

I would read this section...

> **[Logstash Configuration Files | Logstash Reference \[8.12\] | Elastic](https://www.elastic.co/guide/en/logstash/current/config-setting-files.html)**

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [March 3, 2024, 5:32pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/4 "2024-03-03T17:32:41Z")

</div>

OK very interesting. So I confirmed a few things:

I used `path.config: "/etc/logstash/conf.d/my-alerts.conf"` in `pipelines.yml` and now `hits` and `hitsjson` appears, when I run the command `/usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic`.

So basically I thought `*.conf` meant wild card...but when you use the `*`, the `hits/hitsjson` wil be removed, or maybe some `filter` plugins are not supported when used with asterisks.

If I remove `/etc/logstash/pipelines.yml` , and run `/usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic`, then I will get an error

```auto
ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml, Exception: #<Errno::ENOENT: No such file or directory - /etc/logstash/pipelines.yml

```

So basically i need to explicitly specify each`.conf` file one at a time in my `/etc/logstash/pipelines.yml`. I can't use any kind of wildcard if I want to retain my `hits/hitsjson`.

Is that the idea?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 3, 2024, 5:42pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/5 "2024-03-03T17:42:44Z")

</div>

Hmmm re-reading the `*` should work.. let me triple check I am going to test

This should work (I will fix my post above)

`path.config: "/etc/logstash/conf.d/*.conf"`

BUT you are confusing the `*` with your filter they are unrelated.....

Either logstash is reading your conf or it is not... you should be able to see it in the logs...

There is no relation between the path of the pipeline and whether a `filter` is supported...

let me test

@learningelastic What version are you using and what install method.

EDIT I just verified `*.conf` works in pipeline.yml

I think you have another issue..

```auto
- pipeline.id: test
  path.config: "/mypath/logstash-8.12.1/config/conf.d/*.conf"

```

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [March 3, 2024, 5:50pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/6 "2024-03-03T17:50:09Z")

</div>

OK thanks! I am using `logstash 8.12.2`. I ran these commands to install logstash on Ubuntu 22.04

```auto
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -;
sudo apt-get install apt-transport-https;
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list;
sudo apt-get update && sudo apt-get -y logstash

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 3, 2024, 5:55pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/7 "2024-03-03T17:55:10Z")

</div>

What do your logstash startup logs show... when you start without `-f`

make sure the permission are readable on all the files and directories

what happens if you start with

`systemctl start logstash`

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [March 3, 2024, 5:57pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/8 "2024-03-03T17:57:13Z")

</div>

actually, looks like there is something else going on after i repeated another test. I'mg oing to try to investigate a bit more and then post an update

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 3, 2024, 5:59pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/9 "2024-03-03T17:59:57Z")

</div>

So you look like you are on the right path... sorry about the `*`... but that should work...

So if you just want to run by command line instead of `systemctl` I guess I would have just installed the tar.gz... when you install as a package and use `systemctl` there are certain permissions set etc.. etc..

---

<div class="post-metadata">

**Author:** ![learningelastic](https://avatars.discourse-cdn.com/v4/letter/l/958977/32.png) [@learningelastic](https://discuss.elastic.co/u/learningelastic)\
**Post date:** [March 3, 2024, 7:03pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/10 "2024-03-03T19:03:31Z")

</div>

@stephenb

I think I may have figured out what's going on .

My `/etc/logstash/conf.d/` actually has several more file in addition to `my-alerts.conf`. For example, I have another file called `/etc/logstash/conf.d/hello-world.conf` with the contents:

```auto
... etc ...
filter {
  mutate {
    remove_field => ["hits", "hitsjson"]
  }
}
... etc ...

```

When I comment out the `mutate { remove_field => ["hits", "hitsjson"] }` in `hello-world.conf`, then they will not be deleted from the `stdout` of `my-alerts.conf`.

So I guess the conf files are accumulative? Can I have each logstash configuration run in its own process and not affect each other?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 3, 2024, 7:07pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/11 "2024-03-03T19:07:35Z")

</div>

> [@learningelastic](#):
>
> So I guess the conf files are accumulative?

Yes all `.conf` files will be concatenated into 1 giant conf file (the docs do say that but it buried)  
But often folks with BIG conf files break them up into input.conf, filter.conf and output.conf etc which then all get catted together.

> [@learningelastic](#):
>
> Can I have each logstash configuration run in its own process and not affect each other?

Yes then list each as separate /specifically in the pipelines.yml

Which bring us full circle why I thought without `*` as I am very careful which pipelines I want to run... so. my convention is to list separately ...

There are use cases for both

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 31, 2024, 7:08pm UTC](https://discuss.elastic.co/t/why-does-path-settings-delete-events-why-does-f-work-perfectly/354592/12 "2024-03-31T19:08:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.