Logtash Giving : : Expected one of #, input, filter, output at line 1, column 1 (byte 1) Error in syntax checking

Elastic Stack Logstash
1

[WARN ] 2022-03-29 15:12:27.166 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2022-03-29 15:12:27.560 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 1, column 1 (byte 1)
[ERROR] 2022-03-29 15:12:27.577 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Steps That i tried to fix this :
1.I checked each pipelines it is giving configuration valid and OK .
For ex: sudo bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/dmb_prod.conf By checking this it is giving Ok for all pipelines.

So can someone help how i can know that what particular line is giving me that error ....

And also in the error message at last it is giving : Logstash stopped processing because of an error: (SystemExit) exit.

So is it a error that can effect processing of data in ES. I am ablwe to see data of this running pipelines but it is not consistent.
So also wanted to know about it as well...

Thanks

2

What command are you using to start Logstash?

3

Systemctl restart logstash

4

But logstash is continously running in my case . I don't stop and restart it again and again

5

Can you post your pipelines.yml config then?

When you run sudo bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/dmb_prod.conf it's checking that single config.

When you are running as a service it should utilize pipelines.yml and execute those pipeline (which there might be an error in). So the steps would be to look at the pipelines.yml first and then go to any of the pipelines that are being started by that.

6

This file is where you define your pipelines. You can define multiple.

For more information on multiple pipelines, see the documentation:

Multiple Pipelines | Logstash Reference [8.1] | Elastic

  • pipeline.id: dmb_prod_pip_all_org
    path.config: "/etc/logstash/conf.d/dmb_prod_pip_all_org.conf"

##- pipeline.id: rest_dmb_prod_pip_all_backup

path.config: "/etc/logstash/conf.d/rest_dmb_prod_pip_all_backup.conf"

  • pipeline.id: dmb_rest_prod_all
    path.config: "/etc/logstash/conf.d/rest_dmb_prod_pip_all.conf"
    pipeline.workers: 3

###- pipeline.id: rest_dmb_prod_pip_all_org

path.config: "/etc/logstash/conf.d/rest_dmb_prod_pip_all_org.conf"

pipeline.workers: 3

  • pipeline.id: dmb_soc_prod_all
    path.config: "/etc/logstash/conf.d/dmb_soc_prod_pip.conf"
  • pipeline.id: rest_dmb_soc_prod
    path.config: "/etc/logstash/conf.d/rest_dmb_soc_prod_pip_all.conf"

#- pipeline.id: dmb_prod_pip_all_backup

path.config: "/etc/logstash/conf.d/dmb_prod_pip_all_backup.conf"

Above is Pipelines.yml.

And for checking each pipelines individually it is giving configuration ok status. But with just sudo bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/

it is giving that errror

7

for better understandin here is SS :slight_smile:

image
image935×647 18.5 KB

8

The one file you tested dmb_prod.conf isn't in the pipelines.yml file that I see.

What's in here is all the pipeline that will execute when you start Logstash as a service. I would go to each one of these and test. Also I am not quite sure what ## and ### will do. Should just be a single # to comment out a line so it doesn't execute.

I would also clean it up so it only has what you are executing to ensure there is no format or typos with all the comments.

- pipeline.id: dmb_prod_pip_all_org
  path.config: "/etc/logstash/conf.d/dmb_prod_pip_all_org.conf"

- pipeline.id: dmb_rest_prod_all
  path.config: "/etc/logstash/conf.d/rest_dmb_prod_pip_all.conf"
  pipeline.workers: 3

- pipeline.id: dmb_soc_prod_all
  path.config: "/etc/logstash/conf.d/dmb_soc_prod_pip.conf"

- pipeline.id: rest_dmb_soc_prod
  path.config: "/etc/logstash/conf.d/rest_dmb_soc_prod_pip_all.conf"
9

that dmb_prod i used is just fir Example i expressed.

So,

  1. Do i need to run sudo bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/
    this command for each pipelins?
    and also do i need to run it for json filter as well ?

  2. Do i need to remove ### this type of comments in pipelines .yml ?

10
  1. Yes. Test the config on any .conf file that is defined in the pipelines.yml
  2. I would remove those comments.
[FATAL] 2022-03-29 15:12:27.560 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 1, column 1 (byte 1)

This means the first character in one of those files does not start with one of the below.

# or input or filter or output

11

Hi Aaron,

I checked for each pipeline individually but all shown configuration OK for all pipelines inside Pipelines.yml

---> do i also need to check .json file with the sudo bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/
this command ?

Is their any way i can find which particular file is causing this error . Because i checked for each .conf file with same command it is giving configuration ok.

12

When you run it as a service and tail the Logstash log does it give you the same error?

[FATAL] 2022-03-29 15:12:27.560 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 1, column 1 (byte 1)

Or does it look different than what you got on the command line? I am asking because I think if you run it as a service it will include the pipeline.id in the error so you know which one it is.

13

No , In the logstash logs it do not gives any error such as invalid configuration like that.....

14

I am not sure what's not working then. If you start Logstash as a service which part of the logs is indicating an error or failure?

I see the one when you started via command line but as a service is different.

15

I do not start logstash on daily basis.

And also when i check status of logstash at that time also it does not gives any error .....

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.