Unable to read the txt file through Logstash

sukumar.koganti · October 5, 2020, 11:44pm

Hi Guys,

I am trying to read sample EDI log from the below path using logstash .conf file
C:\Users\skkoganti\Documents\log\EDI Log-2019-05-27.txt

this is my sample input file format

input

{

file

{

path => ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27"]

start_position => "beginning"

sincedb_path => "NULL"

}

output

{

stdout {codec => rubydebug}

elasticsearch

{

index=> "edi_logs"

hosts => ["localhost:9200"]

}

but i don't see any index created on my kibana. Can anyone please suggest me on this.

logstash output after i ran

C:\Users\skkoganti\Documents\Elastic\elastic_download\logstash-7.9.1\logstash-7.9.1\config>logstash -f iphone.conf
Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties
[2020-10-05T19:38:20,405][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-10-05T19:38:20,690][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-05T19:38:22,842][INFO ][org.reflections.Reflections] Reflections took 50 ms to scan 1 urls, producing 22 keys and 45 values
[2020-10-05T19:38:25,792][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2020-10-05T19:38:26,024][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-10-05T19:38:26,096][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-10-05T19:38:26,103][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2020-10-05T19:38:26,184][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-10-05T19:38:26,293][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2020-10-05T19:38:26,352][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/skkoganti/Documents/Elastic/elastic_download/logstash-7.9.1/logstash-7.9.1/config/iphone.conf"], :thread=>"#<Thread:0x483fc6a run>"}
[2020-10-05T19:38:26,384][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-10-05T19:38:27,279][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.92}
[2020-10-05T19:38:27,870][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-10-05T19:38:27,946][INFO ][filewatch.observingtail ][main][5d1e9063f71372cd43b3d1a5773212aeb482e63ccd792a234be6d0096907cf78] START, creating Discoverer, Watch with file and sincedb collections
[2020-10-05T19:38:27,952][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-10-05T19:38:28,321][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Badger · October 6, 2020, 2:43am

On Windows that should be "NUL", on UNIX it should be "/dev/null"

sukumar.koganti · October 6, 2020, 12:54pm

Thanks Badger,

I changed to sincedb_path => "NUL"

but after ran again i am getting the same output.

C:\Users\skkoganti\Documents\Elastic\elastic_download\logstash-7.9.1\logstash-7.9.1\config>logstash -f iphone.conf
Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties
[2020-10-06T08:49:01,367][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-10-06T08:49:01,658][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-06T08:49:03,816][INFO ][org.reflections.Reflections] Reflections took 46 ms to scan 1 urls, producing 22 keys and 45 values
[2020-10-06T08:49:06,478][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2020-10-06T08:49:06,706][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-10-06T08:49:06,766][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-10-06T08:49:06,772][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2020-10-06T08:49:06,846][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-10-06T08:49:06,912][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2020-10-06T08:49:06,981][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/skkoganti/Documents/Elastic/elastic_download/logstash-7.9.1/logstash-7.9.1/config/iphone.conf"], :thread=>"#<Thread:0x72e93d12 run>"}
[2020-10-06T08:49:07,007][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-10-06T08:49:07,933][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.93}
[2020-10-06T08:49:08,529][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-10-06T08:49:08,594][INFO ][filewatch.observingtail ][main][ccc290222d51a93cf499be94ceda483818c7289c6274c84d9ca62971ff6a1691] START, creating Discoverer, Watch with file and sincedb collections
[2020-10-06T08:49:08,597][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-10-06T08:49:08,951][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

sukumar.koganti · October 6, 2020, 1:28pm

Hey Badger, it's working now thanks for your help.

path => ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt"]

after given the .txt at end of the file then only it's working.

but i have one more question like i don't see column names on kibana whatever i had in my .txt file.

This is my sample .txt file

logstash output.

[2020-10-06T08:57:04,017][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-10-06T08:57:04,436][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
{
"path" => "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt",
"message" => "2019-05-27 08:59:30|b_GetOtherExcelData_BEEM|Started\r",
"host" => "GTAPIDEV",
"@version" => "1",
"@timestamp" => 2020-10-06T12:57:04.598Z
}
{
"path" => "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt",
"message" => "2019-05-27 13:23:42|ProcessContiGold_contiGOLD|Started\r",
"host" => "GTAPIDEV",
"@version" => "1",
"@timestamp" => 2020-10-06T12:57:04.602Z
}
{
"path" => "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt",
"message" => "2019-05-27 13:54:21|SendContiGOLD_contiGOLD|Started\r",
"host" => "GTAPIDEV",
"@version" => "1",
"@timestamp" => 2020-10-06T12:57:04.605Z

=> i want to read column names aswell, Can you help me badger.

Thanks,
Suresh Koganti.

Badger · October 6, 2020, 1:44pm

Have you tried using a csv filter?

sukumar.koganti · October 6, 2020, 1:51pm

Yes i tried like below, but i am reading .txt file right will it work?

input

{

file

{

path => ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt"]

start_position => "beginning"

sincedb_path => "NUL"

}

filter {

    csv{

    separator => ","

    columns["Time","Application","Message"]

    #autodetect_column_names => true

    }

}

output

{

stdout {codec => rubydebug}

elasticsearch

{

index=> "edi_logs"

hosts => ["localhost:9200"]

}

Badger · October 6, 2020, 2:28pm

I would expect that to work. What happened when you tried it?

sukumar.koganti · October 6, 2020, 2:31pm

when i tied this i am getting like below and i don't see my column names.

                                                                                                                                                            "@version" => "1",
                                                                                                                                                                "host" => "GTAPIDEV",
                                                                                                                                                                "path" => "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt",
"2019-05-27 08:59:33|b_GetOtherExcelData_BEEM|2019 Spring Bookings.xlsx Copied to \\\\pneus.touchette\\documents\\EDI\\BEEM\\Development\\Outbound\\Wholesale\\toSend" => "2019-05-27 13:13:24|SendContiGOLD_contiGOLD|Started",
                                                                                                                                                             "message" => "2019-05-27 13:13:24|SendContiGOLD_contiGOLD|Started\r",
                                                                                                                                                          "@timestamp" => 2020-10-06T14:19:00.237Z

}
{
"@version" => "1",
"host" => "GTAPIDEV",
"path" => "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt",
"2019-05-27 08:59:33|b_GetOtherExcelData_BEEM|2019 Spring Bookings.xlsx Copied to \\pneus.touchette\documents\EDI\BEEM\Development\Outbound\Wholesale\toSend" => "2019-05-27 13:43:15|BuildData_contiGOLD|Started",
"message" => "2019-05-27 13:43:15|BuildData_contiGOLD|Started\r",
"@timestamp" => 2020

Badger · October 6, 2020, 2:40pm

Those are not comma separated values, so I do not understand why you would expect a csv filter to parse them.

sukumar.koganti · October 6, 2020, 2:48pm

Yeah those are pipe (|) separated values and coming from the .txt file like below.

Badger · October 6, 2020, 2:54pm

Maybe you should try

separator => "|"

instead.

sukumar.koganti · October 6, 2020, 3:39pm

Thanks Badger it's working now.

sukumar.koganti · October 6, 2020, 4:46pm

Hi Badger, again...

How to read the latest file based on date and time automatically..

input

{

file

{

path => ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt"]

start_position => "beginning"

sincedb_path => "NUL"

Iker · October 6, 2020, 4:59pm

By default Logstash sorts the files by "last_modified" attribute, you could change this behavior with the "file_sort_by" and "fiel_sort_direction" options.

sukumar.koganti · October 7, 2020, 3:38pm

Thanks Iker, it's working now.

any idea how to fetch the files every week (7 days) from the path.

input

{

file

{

*path => ["//pneus.touchettedocuments/EDI/Logs/Preproduction/txt"]

start_position => "beginning"

sincedb_path => "NUL"

}

filter {

** csv {**

** separator => "|"**

** columns => ["Time","Application","Message"]**

** autodetect_column_names => true**

** autogenerate_column_names => true**

** file_sort_direction => ["asc"]**

** }**

sukumar.koganti · October 7, 2020, 5:28pm

Hi Badger,

When i use this code i am getting below error.

filter {

csv {

separator => "|"

columns => ["Time","Application","Message"]

autodetect_column_names => true

autogenerate_column_names => true

}

mutate {

  convert => {

      "Time" => "Date"

      }

  rename => { "Time" => "Date" }

}

C:\Users\skkoganti\Documents\Elastic\elastic_download\logstash-7.9.1\logstash-7.9.1\config>logstash -f edi_development.conf
Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties
[2020-10-07T13:25:47,702][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [mswin32-x86_64]"}
[2020-10-07T13:25:47,983][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-07T13:25:49,446][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "=>" at line 25, column 8 (byte 445) after filter {\r\n csv {\r\n separator => "|"\r\n columns => ["Time","Application","Message"]\r\n autodetect_column_names => true\r\n autogenerate_column_names => true\r\n }\r\n mutate {\r\n\t convert => {\r\n "Time" => "Date"\r\n }\r\n\t rename => { "Time" => "Date" }\r\n}\r\noutput\r\n{\r\nstdout ", :backtrace=>["C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/java_pipeline.rb:44:in initialize'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/agent.rb:357:in block in converge_state'"]}
[2020-10-07T13:25:49,862][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-07T13:25:54,621][INFO ][logstash.runner ] Logstash shut down.
[2020-10-07T13:25:54,640][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

sukumar.koganti · October 7, 2020, 5:29pm

actually i am trying to convert the columns and along with the data type.

mutate {

  convert => {

      "Time" => "Date"

      }

  rename => { "Time" => "Date" }

}

Badger · October 7, 2020, 6:25pm

You are missing a } to close the filter section.

Also, mutate cannot do a "Date" conversion. You would need a date filter to parse a string into a date.

sukumar.koganti · October 7, 2020, 7:22pm

Hmmm, bit confusing with this

can we write like this.

filter {
csv {
separator => "|"
columns => ["Time","Application","Message"]
autodetect_column_names => true
autogenerate_column_names => true
}
filter {
date {
convert => { "string" => "Date" }
}
mutate {
rename => { "Time" => "Date" }
}
}
}

Badger · October 7, 2020, 7:56pm

You have to tell the date filter what the format of the string is. If yours is 2019-05-27 08:59:30 then you would use

date { match => [ "Time", "YYYY-MM-dd HH:mm:ss" ] }

Topic		Replies	Views
Cannot Index Log Files into Elasticsearch using Logstash Elasticsearch	7	3978	July 6, 2017
Logstash does not do anything with file input but starts successfully Logstash	5	481	April 9, 2019
What is the process to index the .txt and .log files into elasticsearch Elasticsearch	5	10111	July 6, 2017
Logstash-* unable to fetch mapping in kibana Logstash	12	1406	November 15, 2017
ELK Stack: Logstash shows that it's receiving log entries from Filebeat, but Elasticsearch is not creating my index Elasticsearch	9	310	January 23, 2024

Unable to read the txt file through Logstash

Related topics