In RHEL how to run Configuration file along with logstash?

Elastic Stack Logstash
21

Hi @magnusbaeck

In RHEL environment we dont have pipelines.yml file to make changes to create multiple pipelines.

so i did configuration in Logstash.yml file as follows:

#------------ Pipeline Settings --------------

  • pipeline.id: monerrorlog-pipeline
    path.config: "/etc/logstash/conf.d/mon-errorlog.conf"
    pipeline.workers: 1
  • pipeline.id: monserverlog-pipeline
    path.config: "/etc/logstash/conf.d/mon-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl1a-pipeline
    path.config: "/etc/logstash/conf.d/sl1a-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl2a-pipeline
    path.config: "/etc/logstash/conf.d/sl2a-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl2b-pipeline
    path.config: "/etc/logstash/conf.d/sl2b-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl2c-pipeline
    path.config: "/etc/logstash/conf.d/sl2c-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl3a-pipeline
    path.config: "/etc/logstash/conf.d/sl3a-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl3b-pipeline
    path.config: "/etc/logstash/conf.d/sl3b-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl3c-pipeline
    path.config: "/etc/logstash/conf.d/sl3c-serverlogs.conf"
    pipeline.workers: 1
  • pipeline.id: sl3d-pipeline
    path.config: "/etc/logstash/conf.d/sl3d-serverlogs.conf"
    pipeline.workers: 1

#------------ Pipeline Configuration Settings --------------
#Where to fetch the pipeline configuration for the main pipeline

#path.config: /etc/logstash/conf.d/*.conf

#Pipeline configuration string for the main pipeline

#config.string:

#At startup, test if the configuration is valid and exit (dry run)

#config.test_and_exit: false

#Periodically check if the configuration has changed and reload the pipeline
#This can also be triggered manually through the SIGHUP signal

#config.reload.automatic: false

#How often to check if the pipeline configuration has changed (in seconds)

#config.reload.interval: 3s

#Show fully compiled configuration as debug log message
#NOTE: --log.level must be 'debug'

#config.debug: false

#When enabled, process escaped characters such as \n and " in strings in the
#pipeline configuration files.

#config.support_escapes: false
.....

so am i done configuration correct??

let me know if any changes required!

Thanks,
Naveena K N

22

Which version of Logstash have you installed?

23

hi @Christian_Dahlqvist

its Logstash 6.0.0 in RHEL

24

Since 6.x they introduced pipelines configuration. You should be able to see pipelines.yml file in your logstash installation folder. Anyway let me download and see.

25

ya suman...please let me know aftr installing
Thanks,
Naveena K N

26

If it is not there try to create pipelines.yml file in you config folder and start logstash with bin/logstash -r option

27

its in RHEL operating system , so are you sure we can start using bin/logstash -r??

28

Well that am not sure of. Ours is Linux - GNU/Linux. Try to start it in your local machine. Then you can try it in RHEL or anywhere. Or your local itself is RHEL? :stuck_out_tongue:

29

ok will wait for ELK team Response

30

its in RHEL operating system , so are you sure we can start using bin/logstash -r??

The choice of Linux distribution doesn't affect whether the -r option works.

31

pipelines.yml file is not there.!!

shall i add pipelines.yml??

Thanks,
Naveena K N

32

try adding it. There is nothing wrong in trying right.!

33

yes suman, we asked team who are have permissions to upload file in the current working environment...

Thanks,
Naveena

34

What I would suggest or prefered to do is.. Do the testing and all the stuff in your local machine and if the configuration works well go to the higher environment. That way you can save lot of time. Just think about it.

35

yes iam testing it in local also with pipeline.yml having :

pipeline.id: test
pipeline.workers: 2
pipeline.batch.size: 1
path.config: "C:\Users\nakn\Downloads\logstash-6.1.1\bin\config\logs-server.conf"

but when i tried to run logstash as follows:

bin/logstash

im getting error as Failed to read pipelines yaml file !!
why am i getting this error??

Thanks,
Naveena

36

Can we have detailed fail log?

37

ya sure

Capture
Capture.PNG1766×891 35.6 KB

38

I dont see any error here

39

ya but while running in command promt im getting this:

Capture
Capture.PNG649×733 12.4 KB

40

Please attach pipelines file