Logstash is not starting in windows

Elastic Stack Logstash
1

Hey Guys ,
I am using Logstash to send logs to kibana . I have configured logstash in windows instances with Two pipelines in my pipelines.yaml and this is the config that is present in

  1. pipelines.yml

- pipeline.id: my-pipeline_1
  path.config: "C:\logstash\bin\logstash.conf"
  pipeline.workers: 3
- pipeline.id: my-other-pipeline
  path.config: "C:\logstash\bin\logstash1.conf"
  queue.type: persisted

A) I have setup the Task manager to run the execution command at every system startup
This is how it looks like

image

B) But when i reboot the server then I dont see it got started and there is no logs also for the execution not sure what I am making wrong here.

2

You need to use -f to specify path.config (the configuration file name).

3

HI @Badger
Yes it worked after using the -f . But How Im not sure how do i confirm my both the pipelines are running as i have setup multiple pipelines one for my IIS logs and another for my Backup logs
This is what the logs shows

[2021-04-22T02:02:14,163][WARN ][logstash.runner          ] SIGTERM received. Shutting down.
[2021-04-22T02:03:41,915][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-04-22T02:03:42,147][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2021-04-22T02:03:44,889][INFO ][org.reflections.Reflections] Reflections took 63 ms to scan 1 urls, producing 21 keys and 41 values 
[2021-04-22T02:03:48,912][INFO ][logstash.outputs.amazonelasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[************:443/]}}
[2021-04-22T02:03:48,932][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>************:443/, :path=>"/"}
[2021-04-22T02:03:49,969][WARN ][logstash.outputs.amazonelasticsearch][main] Restored connection to ES instance {:url=>"************:443/"}
[2021-04-22T02:03:50,049][INFO ][logstash.outputs.amazonelasticsearch][main] ES Output version determined {:es_version=>7}
[2021-04-22T02:03:50,054][WARN ][logstash.outputs.amazonelasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-04-22T02:03:50,076][INFO ][logstash.outputs.amazonelasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::AmazonElasticSearch", :hosts=>["************"]}
[2021-04-22T02:03:50,095][INFO ][logstash.outputs.amazonelasticsearch][main] Using mapping template from {:path=>nil}
[2021-04-22T02:03:50,137][INFO ][logstash.outputs.amazonelasticsearch][main] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60002, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2021-04-22T02:03:50,605][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[2021-04-22T02:03:51,073][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["c:/logstash/bin/logstash.conf"], :thread=>"#<Thread:0x4fd039e2 run>"}
[2021-04-22T02:03:53,289][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"c:/logstash/data/plugins/inputs/file/.sincedb_0d20cbad6802ed3feeba4d6a5606d787", :path=>["C:/inetpub/logs/LogFiles/W3SVC8/*"]}
[2021-04-22T02:03:53,325][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-04-22T02:03:53,404][INFO ][filewatch.observingtail  ][main][4d1caac663897fbc3eed91da0efd905f706201a116cf573a0163eafc8c99f0d3] START, creating Discoverer, Watch with file and sincedb collections
[2021-04-22T02:03:53,408][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-04-22T02:03:53,803][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
4

Since you want to start logstash when the system starts, It would be better to create a Windows Service using NSSM, you can follow the documentation to do that.

Also, if you use the -f option in the command line it will ignore your pipelines.yml, if you want to use the pipelines.yml file, you should just call logstash.bat in your task and you need just one instance, logstash will use the pipelines.yml to start your pipelines.

5

Hi @leandrojmp
This is what i am getting while running the command

Sending Logstash logs to C:/logstash/logs which is now configured via log4j2.properties
ERROR: Failed to read pipelines yaml file. Location: C:/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2021-04-26T04:12:07,551][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

2.This is what my pipelines.yml has

- 
pipeline.id: pipeline1
  path.config: "C:\logstash\bin\logstash.conf"
  pipeline.workers: 2

- 
pipeline.id: pipeline2
  path.config: "C:\logstash\bin\logstash1.conf"
  queue.type: persisted
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.