Why does `--path.settings` delete events? Why does `-f` work perfectly?

Elastic Stack Logstash
1

When I start logstash with -f, everything works perfectly. If I start logstash with --path.settings, logstash deletes events received from the input. Does anyone know why?

Here's how to reproduce the issue...i start with a success case followed by a failed case.

Scenario 1: -f works perfectly

I have an index called my-alerts with a simple mapping like this:

{
    "properties": {
      "alert_type": {
        "type": "keyword"
      },
      "hits": {
        "type": "keyword"
      }
    }
}

I created a file called /etc/logstash/conf.d/my-alerts.conf like this:

input {
  elasticsearch {
    schedule => "* * * * *"
    hosts => "${ES_HOST}"
    user => "${ES_USER}"
    password => "${ES_PASS}"
    index => "my-alerts"
    query => '{"query":{"bool":{"must":[{"match":{"alert_type":"changes"}}]}}}'
    ssl_enabled => true
    ssl_certificate_authorities => "${ES_CA_FILE}"
    ssl_verification_mode => "full"
  }
}
filter {
  json {
    source => "hits"
    target => "hitsjson"
  }
}
output {
  stdout {}
}

I created a run.sh like this:

ES_HOST="es01"
ES_USER="elastic"
ES_PASS="changeme"
ES_CA_FILE="/etc/certs/es/ca.crt"
export ES_HOST
export ES_USER
export ES_PASS
export ES_CA_FILE

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/my-alerts.conf --config.reload.automatic

When I run the command ~/run.sh everything works perfectly.

Scenario 2: --path.settings deletes the hits/hitsjson fields

However, if I replace the line

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/my-alerts.conf --config.reload.automatic

with this line:

/usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic

then I will not see the events/fields hits or hitsjson. Why does using /usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic delete the hits/hitsjson fields?

Also this is my /etc/logstash/pipelines.yml

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
2

Most likely because then logstash is not using your conf file..

if you Do not use -f logstash will look for the pipelines.yml file for which pipeline to run

Check to see if you have a pipelines.yml if you want logstash to autoload conf.d remove that file

I would read this section...

4

OK very interesting. So I confirmed a few things:

I used path.config: "/etc/logstash/conf.d/my-alerts.conf" in pipelines.yml and now hits and hitsjson appears, when I run the command /usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic.

So basically I thought *.conf meant wild card...but when you use the *, the hits/hitsjson wil be removed, or maybe some filter plugins are not supported when used with asterisks.

If I remove /etc/logstash/pipelines.yml , and run /usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.reload.automatic, then I will get an error

ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml, Exception: #<Errno::ENOENT: No such file or directory - /etc/logstash/pipelines.yml

So basically i need to explicitly specify each.conf file one at a time in my /etc/logstash/pipelines.yml. I can't use any kind of wildcard if I want to retain my hits/hitsjson.

Is that the idea?

5

Hmmm re-reading the * should work.. let me triple check I am going to test

This should work (I will fix my post above)

path.config: "/etc/logstash/conf.d/*.conf"

BUT you are confusing the * with your filter they are unrelated.....

Either logstash is reading your conf or it is not... you should be able to see it in the logs...

There is no relation between the path of the pipeline and whether a filter is supported...

let me test

@learningelastic What version are you using and what install method.

EDIT I just verified *.conf works in pipeline.yml

I think you have another issue..

- pipeline.id: test
  path.config: "/mypath/logstash-8.12.1/config/conf.d/*.conf"
6

OK thanks! I am using logstash 8.12.2. I ran these commands to install logstash on Ubuntu 22.04

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -;
sudo apt-get install apt-transport-https;
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list;
sudo apt-get update && sudo apt-get -y logstash
7

What do your logstash startup logs show... when you start without -f

make sure the permission are readable on all the files and directories

what happens if you start with

systemctl start logstash

8

actually, looks like there is something else going on after i repeated another test. I'mg oing to try to investigate a bit more and then post an update

1 Like
9

So you look like you are on the right path... sorry about the *... but that should work...

So if you just want to run by command line instead of systemctl I guess I would have just installed the tar.gz... when you install as a package and use systemctl there are certain permissions set etc.. etc..

10

@stephenb

I think I may have figured out what's going on .

My /etc/logstash/conf.d/ actually has several more file in addition to my-alerts.conf. For example, I have another file called /etc/logstash/conf.d/hello-world.conf with the contents:

... etc ...
filter {
  mutate {
    remove_field => ["hits", "hitsjson"]
  }
}
... etc ...

When I comment out the mutate { remove_field => [ "hits", "hitsjson" ] } in hello-world.conf, then they will not be deleted from the stdout of my-alerts.conf.

So I guess the conf files are accumulative? Can I have each logstash configuration run in its own process and not affect each other?

11

Yes all .conf files will be concatenated into 1 giant conf file (the docs do say that but it buried)
But often folks with BIG conf files break them up into input.conf, filter.conf and output.conf etc which then all get catted together.

Yes then list each as separate /specifically in the pipelines.yml

Which bring us full circle why I thought without * as I am very careful which pipelines I want to run... so. my convention is to list separately ...

There are use cases for both

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.