Hello,
Input csv looks like :
"","Code","Date","Time","Open","High","Low","Close","Volume"
"1","3IINFOTECH",20150703,"09:16:00",5.55,4.55,4.55,4.55,835
"2","3IINFOTECH",20150703,"09:17:00",5.55,4.55,4.55,4.55,390
Logstash config is :
input {
file {
path => "/Users/sbezgoan/Documents/Elastic/StockData/nse-company-stocks/three.csv"
start_position => "beginning"
sincedb_path => "/Users/sbezgoan/Documents/Tools/logstash-7.5.1/temp.log"
}
}
filter {
csv {
separator => ","
columns => ["SNo","Code", "Date", "Time", "Open","High", "Low", "Close","Volume"]
skip_header => true
}
mutate {
add_field => {"DateTime" => "%{Date} %{Time}"}
}
mutate {
remove_field => ["%{Time}", "%{Date}"]
}
date{
match => ["DateTime" , "YYYYMMdd HH:mm:ss"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "nse-stocks2"
action => "update"
doc_as_upsert => "true"
document_id => "%{[SNo]}"
manage_template => "true"
script => 'if (ctx._source.Open == null) { ctx._source.Open = new ArrayList();} if(ctx._source.Open != null) {ctx._source.Open.add("%{[Open]}");}'
}
stdout {}
}
The aim is if a document with the same ID already exists, then append to the Open field new incoming values creating an array of Open values.
The script doesn't seem to be doing the intended, appreciate any help on how I can achieve Open field mapping as a multi-valued array.
Also, I don't like the fact that I am allowing logstash to manage templates (elastic mappings).
I think the issue is the way I am retrieving the existing _source, but not sure how to fix.