Hi,
I'm checking my logstash logs and see many entries with WARN "Error parsing csv" (80% cases are well parsed). ¿could u help me out? Many thanks in advance.
Furthermore, "date" field is not added to timestamp index.
Logstash conf file:
input {
file {
path => "/home/adminfran/desktop/monitoring/act_final3.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["id tweet","date","author","text","app","id user","followers","following","stauses","location","urls","geolocation","name","description","url_media","type media","quoted","relation","replied_id","user replied","retweeted_id","user retweeted","quoted_id","user quoted","first HT","lang","created_at","verified","avatar","link"]
separator => " " #tab
}
date {
match => ["date","yyyy-MM-dd HH:mm:ss"]
timezone => "UTC"
target => "date"
}
mutate {
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "monitoring_twitter"
}
stdout{codec => rubydebug}
}
Error:
[WARN ] 2019-12-13 13:03:17.881 [[main]>worker1] csv - Error parsing csv {:field=>"message", :source=>"1146480753682898944\t2019-07-03 18:08:06\t@monterhusmore\tRT @Sertemus: Sagen sie mir, wenn ich falsch liege: hatten polizeibeamte, die @KRLS beobachteten, bereits einen gerichtsbeschluss? Wenn nicht, wer erteilt ihnen den auftrag, diese überwachung auf deutschem gebiet durchzuführen? @DerSPIEGEL @nytimes @guardian @BILD @lalsace @washingtonpost https://t.co/XsVtCC0FQi\tTwitter for Android\t2761881400\t2280\t4979\t200492\tNone\thttps://twitter.com/portet_bruguera/status/1146069575462637570\tNone\tMONTERHUSMORE.\tNone\tNone\tNone\tMHP @KRLS "El TJUE ha de resoldre, ha d'intervenir. Alguns ens voldrien silenciats, quiets i tancats. No hem callat, no ens aturarem i no ens rendirem. Deixeu-me dir ben clar: visca Europa i visca Catalunya Lliure" #PersisitimiGuanyarem https://t.co/5HsnXf6pGO\tquote\tNone\tNone\tNone\tNone\t1146069575462637570\tportet_bruguera\tNone\tde\t2014-09-05 10:39:56\tFalse\thttps://pbs.twimg.com/profile_images/1139420260917006336/dUbIjzwm_normal.jpg\thttps://twitter.com/monterhusmore/status/1146480753682898944", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
