spotlight.csv:
ENTITY_ID,ENTITY_TYPE,SPN_FIRM_ID,IA_ID,BRANCH_ID
1323575,AC,101,10503,11002
1323575,AC,101,10503,359006
Expecting Output JSON:
{
"keyInformation": {
"entityId": 1323575,
"entityType": "AC",
"firmId": 101
},
"entitlements": {
"branchId": [ 11002,359006],
"iaId": 10503
}
}
spotlight.conf:
input {
file {
path => "C:/ELK/logstash-7.2.0/config/spotlight.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["ENTITY_ID","ENTITY_TYPE","SPN_FIRM_ID",
"IA_ID","BRANCH_ID"]
}
mutate {
rename => {
"ENTITY_ID"=> "[keyInformation][entityId]"
"ENTITY_TYPE"=> "[keyInformation][entityType]"
"SPN_FIRM_ID"=> "[keyInformation][firmId]"
"IA_ID"=> "[entitlements][iaId]"
"BRANCH_ID"=> "[entitlements][branchId]"
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch{
hosts => [ "localhost:9200" ]
index => "spotlight"
document_id => "%{[keyInformation][entityId]}"
document_type => "_doc"
user => elastic
password => test01
action => "update"
doc_as_upsert => "true"
}
}
Branch Id is different for the same key, need to create a list of branch Id's for the key. Can some help me how to achieve using logstash?