Logstash shut down

Elastic Stack Logstash
1

Hi
I have successfully installed/configured the elasticsearh and kibana both along with the logstash.

i am trying to process the simple .csv file through logstash.conf file but i am facing the below error.

C:\elastic_stack\logstash-7.8.1>logstash -f logstash.conf
Sending Logstash logs to C:/elastic_stack/logstash-7.8.1/logs which is now configured via log4j2.properties
[2020-08-17T11:06:09,515][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-08-17T11:06:09,606][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.8.1", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-08-17T11:06:10,085][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"C:/elastic_stack/logstash-7.8.1/logstash.conf"}
[2020-08-17T11:06:10,108][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2020-08-17T11:06:10,467][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-08-17T11:06:15,401][INFO ][logstash.runner ] Logstash shut down.

Could you please help me on this.

Here is my logstash.conf file ---> I have placed this in both the paths for safe purpose.

(C:\elastic_stack\logstash-7.8.1\config\logstash.conf)
(C:\Users\kogan\Documents\Talend\logstash.conf)

input {

file{

    path => "C:/Users/kogan/Documents/Talend/*.csv"

    start_position => "beginning"

}

}

filter{

csv{

    seperator => ","

    columns => ["id","name","host_id","host_name","neighbourhood_group","neighbourhood","latitude","longitude","room_type","price","minimum_nights","number_of_reviews","last_review","reviews_per_month","calculated_host_listings_count","availability_365"]

}

}

output{

elasticsearch{

    hosts => "http://localhost:9200"

    index => "ab_nyc_2019"

}

stdout {}

}

2

C:/elastic_stack/logstash-7.8.1/logstash.conf does not exist.

Run logstash -f config/logstash.conf to execute the .conf file in your config folder.

3

Thanks for your reply :slight_smile:

I just ran the logstash -f config/logstash.conf

even though i am getting the below error

LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at C:/elastic_stack/logstash-7.8.1/logstash-core/lib/logstash/agent.rb:369
[2020-08-17T11:31:07,666][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[2020-08-17T11:31:07,707][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "C:/elastic_stack/logstash-7.8.1/logstash-core/lib/logstash/agent.rb:369:in `block in converge_state'"]}
[2020-08-17T11:31:07,723][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

4

There is at least 1 CSV at C:/Users/kogan/Documents/Talend/?

input {
  file{
    path => "C:/Users/kogan/Documents/Talend/*.csv"
    start_position => "beginning"
  }
}
output {
  stdout {}
}

What do you see if you run the above?

5

Yeah aaron,

When i ran the simple conf file which you mentioned, seems like it's working fine.

[2020-08-17T11:46:49,744][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-08-17T11:46:50,215][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
C:/elastic_stack/logstash-7.8.1/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"message" => "3,Glenn Oaks Office Supplies,1859 Green Bay Rd.,2,1225.",
"@timestamp" => 2020-08-17T15:46:50.433Z,
"path" => "C:/Users/kogan/Documents/Talend/customer_state.csv",
"host" => "DESKTOP-2M19H9R",
"@version" => "1"
}
{
"message" => "2595,Skylit Midtown Castle,2845,Jennifer,Manhattan,Midtown,40.75362,-73.98377,Entire home/apt,225,1,45,2019-05-21,0.38,2,355",
"@timestamp" => 2020-08-17T15:46:50.518Z,
"path" => "C:/Users/kogan/Documents/Talend/ab_nyc_2019.csv",
"host" => "DESKTOP-2M19H9R",
"@version" => "1"
}
{
"message" => "2,Bill's Dive Shop,511 Maple Ave. Apt. 1B,3,88792",
"@timestamp" => 2020-08-17T15:46:50.430Z,
"path" => "C:/Users/kogan/Documents/Talend/customer_state.csv",
"host" => "DESKTOP-2M19H9R",
"@version" => "1"
}
{

where i can check this output at kibana

6

Either your filter or output to ES is causing an issue then. I would try adding the filter back in and check the output. If that works then add your output to ES and test.

7

Thanks Aaron, It's working fine

8

Hi aaron,

Good evening, i hope you are doing well.

Today i tried to connect SQL Server through logstash with sample sql.conf

here it is file name

input {

jdbc {

path => "C:\elastic_stack\logstash-7.8.1\config\sql.conf" 

jdbc_connection_string => "jdbc:sqlserver://localhost;integratedSecurity=true;"

jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"

jdbc_username => "xxx"

servername => "DESKTOP-2M19H9R"

statement => "SELECT * FROM Persons"

}

}

output {

elasticsearch {

hosts => ["localhost:9200"]

index => "cs_users"

}

}

I am facing the below error, can you please help me

C:\elastic_stack\logstash-7.8.1>logstash -f sql.conf
Sending Logstash logs to C:/elastic_stack/logstash-7.8.1/logs which is now configured via log4j2.properties
[2020-08-18T14:33:46,563][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-08-18T14:33:46,688][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.8.1", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-08-18T14:33:47,315][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"C:/elastic_stack/logstash-7.8.1/sql.conf"}
[2020-08-18T14:33:47,406][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2020-08-18T14:33:47,874][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-08-18T14:33:52,756][INFO ][logstash.runner ] Logstash shut down.

9

That is the line that is telling you what's wrong. When you run the below command to start it you need to tell it where the .conf file is after the -f. Most likely you saved the .conf in the config folder again so you need to tell it to look there for it.

logstash -f config/sql.conf
10

Thanks Aaron,

Yes i have the confif file in the below path is this correct?
C:\elastic_stack\logstash-7.8.1\config

and i ran the below command i got output like this
logstash -f config/sql.conf
Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included the proper driver client libraries in your classpath?
Exception: LogStash::PluginLoadingError
Stack: C:/elastic_stack/logstash-7.8.1/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.5/lib/logstash/plugin_mixins/jdbc/common.rb:36:in load_driver' C:/elastic_stack/logstash-7.8.1/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.5/lib/logstash/inputs/jdbc.rb:275:in run'
C:/elastic_stack/logstash-7.8.1/logstash-core/lib/logstash/java_pipeline.rb:374:in inputworker' C:/elastic_stack/logstash-7.8.1/logstash-core/lib/logstash/java_pipeline.rb:365:in block in start_input'

11

Checkout the documentation.

Here is the sample on there.

input {
  jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
    jdbc_user => "mysql"
    parameters => { "favorite_artist" => "Beethoven" }
    schedule => "* * * * *"
    statement => "SELECT * from songs where artist = :favorite_artist"
  }
}

You can remove the path parameter from yours. Doesn't do anything.

The important part of your error is.

Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. :jdbc_driver_library is not set, are you sure you included the proper driver client libraries in your classpath?

What it is looking for is the jdbc_driver_library parameter. All databases have a driver and what you need to do is download that driver and then reference the path of where the .jar file is like in the sample. Where ever you decide to save that file just do the absolute file path to it in the settings.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.