[Resolved] LS pipeline.id not being honoured in logstash.yml

Elastic Stack Logstash
1

I have a logstash instance running as a docker continer. Since it is using only a single pipeline, I've not bothered configuring the pipeline.yml. Instead, I've defined the pipeline params in the logstash.yml file itself:

path.data: path.data: /usr/share/logstash/data
pipeline.id: log_my_stash
pipeline.batch.size: 250
pipeline.batch.delay: 5
pipeline.workers: 24

When I observe the logstash instance and the pipeline via a separate LS monitoring instance, the name of the pipeline is always "main". Even the logs generated by LS is always having "main" as the pipeline name. What gives? The official doc suggests that pipeline.id is what should be used to change the name of the pipeline.

2

Yes, it's by default. If you want to change it, set pipeline.id as you did.
In log will something like this:
[INFO ][logstash.javapipeline ][ls] Starting pipeline {:pipeline_id=>"ls", "pipeline.workers"=>...

No sure what do you expect.

3

What I expect is that the name of the pipeline be "log_my_stash" and not "main". And yet, I see "main" printing all over the logs.

4

Have you restated LS?

5

A gazillion times, since I've been making changes to the pipelines. Just want to make sure i'm not missing something trivial

6

Have you check that you have the same param already somewhere in logstash.yml?

Try to set log.level: debug and see what path.settings is set to.
I didn't have any issue after changing. Restart and that's all folks.

Every file in the host directory ~/pipeline/ will then be parsed by Logstash as pipeline configuration.

7

Please share your logs from logstash and also how you are starting your logstash container.

Are you using docker compose or just cli?

If the pipeline id name is not changing, it is probably not reading your updated logstash.yml file.

8

It's a simple docker-compose.yml file with the service. I know that it's reading the pipeline files. As for the logstash.yml, I believe it's reading the file since I was experimenting with the batch and timeout parameters. A restart promptly applys those changs.

9

Please share the your docker compose and fresh logstash logs.

Not possible to troubleshoot without those information.

10

Thx. I had a gut feel it was something trivial. The docker container had a pipeline.yml which had main in it. So it took that into account. Whatever was not part of pipeline.yml, it refered to my logstash.yml. Hence I was able to set all my batch and workers, but just not the pipeline id.

Issue stands resolved

2 Likes