Logstash not running

Elastic Stack Logstash
1

I've been trying and trying to get logstash to run but it keeps coming up with the same error:
ERROR: Pipelines YAML file is empty. Location: /Users/bollampally.r/Downloads/sre/logstash-6.4.0/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2018-08-31T18:42:08,591][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

and when i change the pipelines file and add input, filter and output data it says pipeline array Pipelines YAML file must contain an array of pipeline configs

Please advise ASAP

2

and when i change the pipelines file and add input, filter and output data it says pipeline array Pipelines YAML file must contain an array of pipeline configs

The pipelines.yml file is not supposed to contain inputs, filters, and outputs. See Multiple Pipelines | Logstash Reference [8.11] | Elastic.

I believe Logstash will look for pipelines.yml when it's not given another configuration via the -e or -f command-line options and there's no path.config setting in logstash.yml.

3

Hey Magnus,

Thank you so much for the reply but can you elaborate on the command line options because I run logstash using the file itself on my mac. Can you give me more details on the
-e and -f command line options?

Appreciate it

4

Those options are described here: https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html

6

The current directory isn't in the search path so if you want to run a script or program there you need to be explicit about it. In this case is means ./logstash -f mypipeline.conf.

7

I ran that code and it says successfully started logstash API but I can't seem to visualize the data in Kibana. I'm assuming there is code you have to run in elasticsearch to connect it to kibana. Can you direct me to it for a mac? I really appreciate all the help.

This is my config file ATM, I need the code just like the logstash one to connect elasticsearch to kibana.

input {

file {

path => “/Macintosh HD/Users/bollampally.r/Downloads/sre/drugdata1.csv“

start_position => "beginning"

sincedb_path => "/dev/null"

}

}

filter {

csv {

  separator => ","

  #Drug Name, Drug Info                         

 columns => [“drug name”, “drug info”]

}

}

output {

elasticsearch {

 hosts => "[http://localhost:9200](http://localhost:9200/)"

 index => “mypharmdrugdata”

}

stdout {}

}

8

also i need logstash to bind to port 9200 but its binding to 9600. Can you tell me how to change this

9

I'm assuming there is code you have to run in elasticsearch to connect it to kibana.

Kibana connects to ES, not the other way around.

Your configuration looks okay, provided there aren't any typos in the filename etc. I suggest you temporarily comment out your elasticsearch output and use a stdout { codec => rubydebug } output to just dump the raw events to stdout. Are you getting anything then? If yes we can look into the ES setup. If no we know the problem is with the reading of the file.

10

also i need logstash to bind to port 9200 but its binding to 9600.

Why do you think you need Logstash to bind to port 9200? That's what ES normally uses.

11

Everytime I run logstash it disregards my config file and comes up with the error Rohits-MBP:bin bollampally.r$ ./logstash -f mypharmconfig.conf
Sending Logstash logs to /Users/bollampally.r/downloads/sre/logstash-6.4.0/logs which is now configured via log4j2.properties
[2018-09-03T16:48:53,662][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-03T16:48:54,512][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.0"}
[2018-09-03T16:48:54,637][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/Users/bollampally.r/Downloads/sre/logstash-6.4.0/bin/mypharmconfig.conf"}
[2018-09-03T16:48:54,646][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-09-03T16:48:55,024][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

But when I change the name to (mypharmconfig.conf.rtf) which is the full name its coming up with a different errorRohits-MBP:bin bollampally.r$ ./logstash -f mypharmconfig.conf.rtf
Sending Logstash logs to /Users/bollampally.r/downloads/sre/logstash-6.4.0/logs which is now configured via log4j2.properties
[2018-09-03T16:57:39,374][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-03T16:57:40,468][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.0"}
[2018-09-03T16:57:41,017][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ", :backtrace=>["/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:157:ininitialize'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state'"]}
[2018-09-03T16:57:41,510][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Also how would I send the data from logstash to elasticsearch with them being on different ports? When I go to kibana right now it says it cant find any data in elasticsearch and im assuming its because logstash is not sending it to elasticsearch.

12

There's a syntax problem in your configuration file. Why does its name end with .rtf? Surely you're not trying to feed Logstash an RTF document?

Also how would I send the data from logstash to elasticsearch with them being on different ports?

That's not a problem. TCP/IP doesn't work like you think it works.

When I go to kibana right now it says it cant find any data in elasticsearch and im assuming its because logstash is not sending it to elasticsearch.

So far I've seen no evidence that Logstash is even starting up with the right configuration so it's too early to worry about Elasticsearch.

1 Like
13

[2018-09-07T11:13:43,289][ERROR][logstash.filters.csv ] Unknown setting 'seperator' for csv
[2018-09-07T11:13:43,306][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/config/mixin.rb:86:in config_init'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/filters/base.rb:126:ininitialize'", "org/logstash/plugins/PluginFactoryExt.java:58:in filter_delegator'", "org/logstash/plugins/PluginFactoryExt.java:226:inplugin'", "org/logstash/plugins/PluginFactoryExt.java:166:in plugin'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:71:inplugin'", "(eval):12:in <eval>'", "org/jruby/RubyKernel.java:994:ineval'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:49:in initialize'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state'"]}
[2018-09-07T11:13:43,737][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

The error here cam up as unknown separator.

14

After removing it =>

Sending Logstash logs to /Users/bollampally.r/downloads/sre/logstash-6.4.0/logs which is now configured via log4j2.properties
[2018-09-07T11:15:51,920][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-07T11:15:52,664][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.0"}

[2018-09-07T11:15:55,269][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2018-09-07T11:15:55,911][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}

[2018-09-07T11:15:55,926][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}

[2018-09-07T11:15:56,248][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}

[2018-09-07T11:15:56,338][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}

[2018-09-07T11:15:56,343][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}

[2018-09-07T11:15:56,385][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}

[2018-09-07T11:15:56,405][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

[2018-09-07T11:15:56,431][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

[2018-09-07T11:15:56,868][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::File start_position=>"beginning", path=>["\\Users\\bollampally.r\\Downloads\\sre\\drugdata1.csv"], id=>"a2a5ab0a039cb59d481c4dbab4d4e4ac39ad2a924d4a91f34c140933f92bfb4e", sincedb_path=>"/dev/null", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_bd7a63d2-26ed-419b-a67d-e3e00a87c913", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc">", :error=>"File paths must be absolute, relative path specified: \Users\bollampally.r\Downloads\sre\drugdata1.csv", :thread=>"#<Thread:0x4f0a1cef run>"}

[2018-09-07T11:15:57,397][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<ArgumentError: File paths must be absolute, relative path specified: \Users\bollampally.r\Downloads\sre\drugdata1.csv>, :backtrace=>["/Users/bollampally.r/downloads/sre/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.5/lib/logstash/inputs/file.rb:269:in block in register'", "org/jruby/RubyArray.java:1734:ineach'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.5/lib/logstash/inputs/file.rb:267:in register'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:241:inregister_plugin'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:252:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:252:in register_plugins'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:395:instart_inputs'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:293:in start_workers'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:199:inrun'", "/Users/bollampally.r/downloads/sre/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:159:in `block in start'"], :thread=>"#<Thread:0x4f0a1cef run>"}

[2018-09-07T11:15:57,429][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[2018-09-07T11:15:57,830][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Please advise ASAP

15

Here's the error message:

File paths must be absolute, relative path specified: \Users\bollampally.r\Downloads\sre\drugdata1.csv

16

can you tell me whats wrong with my file path

17

Try including a drive letter, i.e. C:\Users\bollampally.r\Downloads\sre\drugdata1.csv or whatever the full path might be.

But... wait. Is this Mac OS? In that case you don't need a drive letter, but you do need to use forward slashes and not backslashes.

18

Can you tell me the code to shut down logstash on a mac
my logstash started running but it came with the error that the mappings were wrong and it had two inputs for the doc so I changed it but now it says there are two instances of logstash running.

Please advise

This is mapping
{
"mypharmdrugdata": {
"aliases": {},
"mappings": {
"_doc": {
"properties": {
"drug_info": {
"type": "text"
},
"drug_name": {
"type": "text"
}
}
}
},
"settings": {
"index": {
"creation_date": "1535605490897",
"number_of_shards": "1",
"number_of_replicas": "1",
"uuid": "DTfjF5ZGT6KUDFjBkVn-9A",
"version": {
"created": "6040099"
},
"provided_name": "mypharmdrugdata"
}
}
}
}

19

Hey @magnusbaeck could you please advise.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.