I'm trying to upload data from redshift into elastic search.
I exported some data from redshift, so now it's just a CSV file, so I thought using logstash with the CSV filter will work.
It looks like it's working, the sincedb file is reaching the end of the file, and after that (when I run logstash locally with the debug flag) I see these lines being printed over and over and it looks as if it will never stop:
_globbed_files: /Users/bla/Downloads/0001*: glob is: ["/Users/bla/Downloads/0001_part_00"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"} Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"} Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"} Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"458", :method=>"flush"}
I checked the ES index and it wasn't updated.
I figure I'm doing something wrong with my config file so here it is:
`input {
file {
path => "/Users/bla/Downloads/0001*"
start_position => "beginning"
ignore_older => 0
}
filter {
csv {
columns => [sessionid, useridentify, firstproject, impressionproject, browsertype, browserversion, os, osversion, oslanguage, devicetype, devicevendor, devicemodel, deviceid, location, screenres, searchengine, bandwidthmeasure, interludeplayer, interludeplayerversion, interludeplayerengine, publisherid, context, referrer, clustername, treehouse_buttonclickcount, pathid, projectcount, desktopviewcount, iosviewcount, andviewcount, sharecount, replaycount, viewcount, viewendcount, interactioncount, noninteractioncount, subscribecount, subscribedisplaycount, subscribeentercount, channelswapcount, ekosubdisplaycount, ekosubintentcount, ekosubsuccesscount, achievementcount, linkoutcount, adskips, adcompletes, eventcount, mineventtime, maxeventtime, minusertime, maxusertime, sessiontotaltime, domloadingtime, domcompletetime, appreadytime, playerloadedtime, city, region, country, latitude, longitude, subscribesource]
convert => {
"mineventtime" => "date_time"
"maxeventtime" => "date_time"
"minusertime" => "date_time"
"maxusertime" => "date_time"
"browserversion" => "integer"
"treehouse_buttonclickcount" => "integer"
"projectcount" => "integer"
"desktopviewcount" => "integer"
"iosviewcount" => "integer"
"andviewcount" => "integer"
"sharecount" => "integer"
"replaycount" => "integer"
"viewcount" => "integer"
"viewendcount" => "integer"
"interactioncount" => "integer"
"noninteractioncount" => "integer"
"subscribecount" => "integer"
"subscribedisplaycount" => "integer"
"subscribeentercount" => "integer"
"channelswapcount" => "integer"
"ekosubdisplaycount" => "integer"
"ekosubintentcount" => "integer"
"ekosubsuccesscount" => "integer"
"achievementcount" => "integer"
"linkoutcount" => "integer"
"adskips" => "integer"
"adcompletes" => "integer"
"eventcount" => "integer"
"sessiontotaltime" => "integer"
"nodestarts" => "integer"
"nodeends" => "integer"
"latitude" => "float"
"longitude" => "float"
}
separator => "|"
remove_field => ["message"]
skip_empty_columns => true
}
}
output {
amazon_es {
action => "index"
index => "sessions"
document_type => "session"
document_id => "%{sessionid}"
hosts => ["our-host-in-amazon.com"]
region => "our-region"
flush_size => 50
idle_flush_time => 5
protocol => "https"
template => "../session_mapping.json"
max_retries => 5
codec => "json"
// All the keys are here too.
}
}
`
Thanks for the help 