Running sample text file thorough logstash

This is my sample file

FName,Lname,sal,location
Suresh,Koganti,6000,Montreal
Denis,Gallagher,7000,Quebec

and this is my conf file

input {

file{

    path => "C:\Users\skkoganti\Documents\log\users.txt"

    start_position => "beginning"

}

}

filter {

    text{

autodetect_column_names => true

}

}

output {

stdout {

codec => rubydebug

}

}

i am getting the below error.

C:\Users\skkoganti\Documents\log>logstash -e 'input{stdin{}}output{stdout{}}'
Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties
[2020-10-02T12:25:15,007][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-10-02T12:25:15,264][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-02T12:25:16,063][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)", :backtrace=>["C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/java_pipeline.rb:44:in initialize'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/agent.rb:357:in block in converge_state'"]}
[2020-10-02T12:25:16,430][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-02T12:25:21,506][INFO ][logstash.runner ] Logstash shut down.
[2020-10-02T12:25:21,525][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

If you specify -e it will not use the configuration file.

I cannot imagine why you would get that error with that command line.

Do not use backslash in the path option of a file input, it is treated as an escape. Use / instead.

Thanks for quick reply,

i am getting the same error again.

C:\Users\skkoganti\Documents\log>logstash -f koganti.conf
Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties
[2020-10-02T13:22:58,927][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-10-02T13:22:59,185][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-02T13:22:59,853][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"C:/Users/skkoganti/Documents/log/koganti.conf"}
[2020-10-02T13:22:59,872][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2020-10-02T13:23:00,238][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-02T13:23:05,116][INFO ][logstash.runner ] Logstash shut down.
[2020-10-02T13:23:05,139][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

No, that is a different message. It is not finding the file.

I am getting the different error this time

C:\Users\skkoganti\Documents\log>logstash -f koganti.conf
Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties
[2020-10-02T13:57:06,266][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-10-02T13:57:06,534][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-02T13:57:08,496][INFO ][org.reflections.Reflections] Reflections took 44 ms to scan 1 urls, producing 22 keys and 45 values
[2020-10-02T13:57:09,320][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/filters/txt>, :path=>"logstash/filters/txt", :type=>"filter", :name=>"txt"}
[2020-10-02T13:57:09,337][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any filter plugin named 'txt'. Are you sure this is correct? Trying to load the txt filter plugin resulted in this error: no such file to load -- logstash/filters/txt", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:82)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.ELK.logstash_minus_7_dot_9_dot_1.logstash_minus_7_dot_9_dot_1.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/java_pipeline.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.ELK.logstash_minus_7_dot_9_dot_1.logstash_minus_7_dot_9_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.ELK.logstash_minus_7_dot_9_dot_1.logstash_minus_7_dot_9_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.ELK.logstash_minus_7_dot_9_dot_1.logstash_minus_7_dot_9_dot_1.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/agent.rb:357)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Unknown Source)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/agent.rb:370
[2020-10-02T13:57:09,349][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[2020-10-02T13:57:09,388][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/agent.rb:370:in `block in converge_state'"]}
[2020-10-02T13:57:09,410][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

You are loading a non-existent filter. In your first post you had

filter {
    text{
        autodetect_column_names => true
    }
}

Did you mean that to be csv?

No, i am actually running the txt file. Later i changed on conf file as per my requirement.

input {

file{

    path => "C:/Users/skkoganti/Documents/log/users.txt"

    start_position => "beginning"

}

}

filter {

    txt{

        separator => ","

        columns => ["FName","Lname","sal","location"]

        autodetect_column_names => true

}

mutate {convert => ["FName", "string"]}

mutate {convert => ["Lname", "string"]}  

mutate {convert => ["sal", "integer"]} 

mutate {convert => ["location", "string"]} 

}

output {

stdout {

codec => rubydebug

}

}

There is no such thing as a txt filter. It looks like you are trying to configure a csv filter. The name of the filter is based on its functionality. It has nothing to do with the name of the file that you are reading.

Thanks Badger,

I don't know how to read sample text file but i want to read my log files which my ETL tool is generating log files. at least i need to test txt file through logstash.

Please help me on this or if you sample conf file please share me.

Thanks,
Suresh Koganti.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.