Thx. I had a gut feel it was something trivial. The docker container had a pipeline.yml which had main in it. So it took that into account. Whatever was not part of pipeline.yml, it refered to my logstash.yml. Hence I was able to set all my batch and workers, but just not the pipeline id.
Issue stands resolved