Setting up logstash config file question

[jason_smith]

Hello Good folks,

New to ELK stack. I have setup Elasticsearch and Kibanan on my windows 8 box and practice index data into elasticsearch from Kibana. Now I want to index data thru Logstash.

I downloaded logstash (ZIP file) and unzipped it. I have a CSV file that I want to index into Elasticsearch. This is the config file I created. I have uploaded that config file as a JPG file.

Now I opened up command prompt (as admin) and typed the following command. I am getting "Unkown command error". Do I need to run anything before I run this command?

           C:\Downloads\ELK Stack\logstash-6.2.2\bin>logstash -f C:\Downloads\ELK Stack\log
            stash-6.2.2\data\logstash-cars.config
            ERROR: Unknown command 'Stack\logstash-6.2.2\data\logstash-cars.config'

[wwalker (Walker)]

This is an issue not unique to Logstash but applies to most command lines...its a simple mistake but should make complete sense when I point it out, lol.

Each command you issue can be followed by an argument/option. Typically, a space is used to separate these two, just like you did with Logstash -f. So what happens when your path has a space in it? The command line thinks your issuing an additional argument or command. Look at your path, locate the space and you'll see that the error matches up with it perfectly.

The fix is straight-forward, encase the path in double quotes logstash -f "C:\Downloads\ELK Stack\logstash-6.2.2\data\logstash-cars.config"

[jason_smith]

Thank You walker. I will try.

Dumb me!! Too much into learning ELK stuff and forgetting basics.

[jason_smith]

Waker,
I issued the command:
logstash -f "C:\Downloads\ELK Stack\logstash-6.2.2\data\logstash-cars.config"
This time, I see in the logs that Logstart started on port 9600. But errors out with this message.
It is a simple CSV file that looks like this. Is there anything that I am missing in the logstash-cars.config file? I have the image of the config file in the original post.

Thank You

                      			[2018-03-09T08:44:15,914][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
		[2018-03-09T08:44:16,091][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
		:exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 3, 
		column 61 (byte 76) after input {\n\tfile{\n\t\tpath=>\"C:\\Downloads\\ELK Stack\\logstash-6.2.2\\data\\car.csv\"", 
		:backtrace=>["C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", 
		"org/jruby/RubyArray.java:2486:in `map'", "C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:11:in 
		`compile_sources'", "C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "
		C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", 
		"org/jruby/RubyArray.java:1734:in `each'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:90:in `execute'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", 
		"C:/Downloads/ELK Stack/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

[jason_smith]

I looked at logstash.yml file. All the commands are commented. Is there anything that I should uncomment in that file?

[wwalker (Walker)]

A couple items here:

This is a good starting point for setting up the Elastic Stack:
https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html

For the logstash.yml, here's a list of the settings available:
https://www.elastic.co/guide/en/logstash/6.2/logstash-settings-file.html

I prefer to set the following on mine:
node.name
config.reload.automatic (This and the following allow you to make changes to the pipelines configs without restarting the service)
config.reload.interval
http.host
http.port
log.level (This is especially useful when troubleshooting issues)

In addition, if you'd like to use the disk to buffer events prior to processing, useful if you have scenarios in which your input data may outpace the speed that the data can be processed and need more space than RAM can provide, set the following minimum:
queue.type
path.queue
queue.max_bytes

For the error message you posted, there's usually a helpful bit on the first couple lines of the message. In this case, we see:

[2018-03-09T08:44:16,091][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
		:exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 3, column 61

So on the first line we see it failed to execute an action related to pipeline creation. Since you didn't assign an ID to your pipeline (not necessary unless you have multiple pipelines), it's tagged as 'main'. On the second line it tells you, on line 3, column 61 it expected to see one of three symbols; #, {, or } but they weren't present. So your problem is that you've got the wrong syntax on line 3, column (aka character) 61.

To make reading and creating your pipeline easier, I recommend using a proper editor like Notepad++.

[jason_smith]

Something is seriously wrong with my setup looks like.

I removed the cars.csv input file from the location. I still get same error. Shouldn't it tell that the source file is missing?

[wwalker (Walker)]

Look at my previous response, and then look at your pipeline, the error is a single character that shouldn't be there, lol.

[jason_smith]

I modified the CSV file. I removed all columns and have only 2 column in the input CSV file (maker and model). And I changed the logstash-cars.config file to reflect this change. still same error.

That shows something not related to the source file CSV at all.

[wwalker (Walker)]

@jason_smith

I modified the CSV file. I removed all columns and have only 2 column in the input CSV file (maker and model). And I changed the logstash-cars.config file to reflect this change. still same error.

That shows something not related to the source file CSV at all.

Dude....slow down....read my responses...I've given you the rod and reel.

[jason_smith]

Ok Thank You for your time Walker.

[jason_smith]

I removed all the data in cars.csv file. Now I have only 2 colums (maker, model) with 3 rows of data in the csv file.

It still complains about column 62 . Only 2 columns exist. It almost looks like logstash is looking into cache where errors are stored. If there is something like that, how can I flush that cached errors?

[2018-03-09T10:34:49,186][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 3, column 62 (byte 77) after input {\n\tfile{\n\t\tpath=>"C:\Downloads\ELK Stack\logstash-6.2.2\data\cars.csv\

[wwalker (Walker)]

For real bro...read my response...you're looking in the wrong place.

[jason_smith]

I read your response again and again.

Here is the issue. Logstash is storing the original error and displaying same thing again and again.
Here is the kicker. I changed the name of the name and location of the config file and the source file as well.

I get same error and it tries to read from cars.csv file when infact that cars.csv file is non-existent at this point.

Logstash clearly is storing somewhere errors and showing again and again. I even closed the current session and re-launched new command prompt session. Let me ask others.

Thank you Walker. You have been very kind

[wwalker (Walker)]

The problem is not with your csv file the problem is with your logstash pipeline. As the error states, it's line 3, column (AKA character) 61. Specifically, you have a / at the end of like 3, that's not supposed to be there.

[jason_smith]

i have never seen anyone so patiently replying.

My upcoming job is in ELK stack. I have to learn before I join.

Thank you