i'm working with logstash the current version 7.6
i have 44 columns , 4 of them are date but contains nothing
i declared them as a data it's not accepted
i didn't dclare them still not accepted ( cuz i know that it's taken as a string default)
so as a solution i'm trying to force them as a string
i tried this code but an error is shwon help
input{
file{
path => "/home/user/data/test-536.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter{
csv{
separator => ","
columns => ["Supervision Period Warning Played Flag",.......,"Community Id 3",]
}
mutate {
convert => {
"Supervision Period Warning Played Flag" => "integer"
.............
"Community Id 3" => "integer"
}
}
mutate {
convert => { "Negative Balance Barring Start Date" => "string" }
convert => { "Last Service Fee Deduction Date" => "string"}
}
mutate {
add_field => {"D"=> ["2020-01-01"]}
}
date {
match => [ "Supervision Period Expiry Date", "YYYY-MM-dd" ]
target => "Supervision Period Expiry Date"
}
Try commenting out the second mutate{}, where you convert date fields to string, and all the date{} plugin instances, and see what you get from stdoutput{}.
You can also comment out your elasticsearch{} output for now (no need to send anything there until you get it right).
I'm thinking you won't need this instance of mutate{} because your data is already string out of the csv{} filter. Then you'll have to "protect" your date{} instances with if statements to check for empty fields. But I'd start, as I suggested above, by checking what you get without filtering the data.
Exactly. The date{} plugin will "crash" if the field is empty, so you have to wrap it inside an if statement that will ensure date{} is called only iff the field is not empty.
well i'm trying to find a condition on empty fields but nothing is working
i found
csv {
source => "message"
}
if ! [""] {
mutate {
update => { "" => " " }
}
}
i tried it but it's not working
i found the solution ( add the ruby filter to see if it's empty field it will remove it else if it's full it will keep it )
input{
file{
path => "/home/user/data/test-536-Copie.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter{
csv{
separator => ","
columns => ["Service Fee Period Warning Played Flag",........,"Community Id 3"]
}
ruby {
code => "
hash = event.to_hash
hash.each do |k,v|
if v == nil
event.remove(k)
end
end
"
}
mutate {
convert => {
"Supervision Period Warning Played Flag" => "integer"
..........
"Community Id 3" => "integer"
}
}
mutate {
add_field => {"D"=> ["2020-01-01"]}
}
date {
match => [ "Supervision Period Expiry Date", "YYYY-MM-dd" ]
target => "Supervision Period Expiry Date"
}
}
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.