Thank you very much for the explanation Magnus
So it will never work because gsub has higher order than copy?
I've tried following the order you posted and used rename instead of copy since it's higher. It appears to be working but the type in kibana is set to number instead of string. Do I need to delete the index and data before it will pick up the new type?
{
"netflow" => {
"output_snmp" => 2,
"dst_as" => 0,
"dst_mask" => 17,
"in_pkts" => 14,
"ipv4_dst_addr" => "192.168.192.105",
"src_tos" => 0,
"first_switched" => "2017-10-25T21:49:59.999Z",
"flowset_id" => 257,
"l4_src_port" => 443,
"ipv4_next_hop" => "192.168.199.2",
"src_mask" => 24,
"version" => 9,
"flow_seq_num" => 1628119,
"ipv4_src_addr" => "192.168.101.12",
"in_bytes" => 8257,
"protocol" => 6,
"last_switched" => "2017-10-25T21:49:59.999Z",
"input_snmp" => 5,
"tcp_flags" => 27,
"flow_sampler_id" => 0,
"l4_dst_port" => 59148,
"src_as" => 0
},
"@timestamp" => 2017-10-25T21:50:00.000Z,
"@version" => "1",
"host" => "192.168.199.1",
"type" => "netflow",
"tags" => [
[0] "netflow",
[1] "NDC",
[2] "Cisco 2901 Router"
],
"direction" => "0 - Ingress"
}
[2017-10-26T09:02:15,564][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.10.25", :_type=>"netflow", :_routing=>nil}, 2017-10-25T22:02:31.000Z 192.168.199.1 %{message}], :response=>{"index"=>{"_index"=>"logstash-2017.10.25", "_type"=>"netflow", "_id"=>"AV9VjrB5cftdLdKJKfte", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [direction]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: "1 - Egress""}}}}}
EDIT:
After deleting the index it picked up the right type so seems all good, so it does work
I'm still trying to find out if I can actually keep the original field and create the second one