I am trying to write an output CSV file that contains two of the four columns from an input CSV file. It appears that the input file is correctly parsed and mapped. The output file is created but nothing gets written to it. It is empty.
Here is my config file:
input {
file { path => "/opt/logstash/testdata/testinput1.txt"
sincedb_path => "/opt/logstash/perl-scripts/sincedb3"
start_position => "beginning"
}
}
filter {
csv {
columns => ["patientid", "name", "health", "concerns"]
separator => ","
}
}
output {
stdout { codec => rubydebug }
file { path => "/opt/logstash/testdata/testoutput1.txt" }
csv {
path => "/opt/logstash/testdata/nameout.csv"
fields => ["patientid", "name"]
csv_options => {"col_sep" => "," "row_sep" => "\r\n"}
}
}
The STDOUT output shows:
{
"message" => [
[0] "12345, Joe Blogs, great health, no worries"
],
"@version" => "1",
"@timestamp" => "2015-09-10T22:24:14.068Z",
"host" => "sem-catalog",
"path" => "/opt/logstash/testdata/testinput1.txt",
"patientid" => "12345",
"name" => " Joe Blogs",
"health" => " great health",
"concerns" => " no worries"
}
Opening file {:path=>"/opt/logstash/testdata/nameout.csv", :level=>:info}
{
"message" => [
[0] "12346, Karen Feeding, good health, some worries"
],
"@version" => "1",
"@timestamp" => "2015-09-10T22:24:14.070Z",
"host" => "sem-catalog",
"path" => "/opt/logstash/testdata/testinput1.txt",
"patientid" => "12346",
"name" => " Karen Feeding",
"health" => " good health",
"concerns" => " some worries"
}
. . . .
Here are a few sample records from the input file. . . .
12341, Robin Banks, great health, no worries
12342, Willy Show, good health, some worries
12343, Lotta Troubles, great health, no worries
12345, Izzy Breathin, good health, some worries
The output should look like:
12341, Robin Banks
12342, Willy Show
12343, Lotta Troubles
12345, Izzy Breathin
UPDATE: The output was getting cached. Upon hitting CTRL-C to terminate logstash, the output file gets written as expected. Also, the default 'plain' codec works best for me. The new config file is:
input {
file { path => "/opt/logstash/testdata/testinput1.txt"
sincedb_path => "/opt/logstash/perl-scripts/sincedb3"
start_position => "beginning"
}
}
filter {
csv {
columns => ["patientid", "name", "health", "concerns"]
separator => ","
}
}
output {
stdout { codec => rubydebug }
csv {
path => "/opt/logstash/testdata/nameout.csv"
fields => ["patientid", "name"]
}
}
PROBLEM SOLVED