Replace the existing index with new data

Elastic Stack Logstash
1

Hi,

I have the index name called "pattwo" ,wanted to update the old data with new data for this index.without creating the new index.
My input file is having two records which is already indexed and the records in this file now updated so need to keep these updated data in above index without creating new index.

Please help me out,I tried but still index is showing old data instead of new data.

Please check my logstash configuration file and csv data.
old CSV data indexed
S2_CR_Browse_iPhone_T22_Click “Cruise&Stay" link from cruise menu,1394,101,92
S2_CR_Browse_desktop_T2_Destinations View All,2785,354,288

New CSV data need to be indexed
S2_CR_Browse_iPhone_T22_Click “Cruise&Stay" link from cruise menu,1395,101,92
S2_CR_Browse_desktop_T2_Destinations View All,2785,354,288

input {
file {
type => "jmetern"
path => ["D:/pat20.csv"]
start_position => "beginning"
}
}

filter {
if([message]=~"sampler_label" or [message]=~"TOTAL")
{
drop{}
}else{
csv {
columns => ["label","SampleCount"]
skip_empty_columns => true
quote_char => "'"
}
}
}
output {
elasticsearch {

  document_id => "%{label}"
   template => "D:/apache-jmeter-2.13/bin/jmeter-mapping.json"
   template_name => "pattwo"
   template_overwrite => false
   hosts => "localhost"
   index => "pattwo"
}

}

Here label field is having unique value so I am referring that as document id.

hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "pattwo",
"_type": "jmetern",
"_id": "\"S2_CR_Browse_iPhone_T22_Click \x93Cruise&Stay\"\" link from cruise menu\"",
"_score": 1,
"_source": {
"message": "\"S2_CR_Browse_iPhone_T22_Click \x93Cruise&Stay\"\" link from cruise menu\",1394,101,92\r",
"@version": "1",
"@timestamp": "2016-07-19T17:19:11.983Z",
"path": "D:/pat20.csv",
"host": "BG3WS0057",
"type": "jmetern",
"label": "\"S2_CR_Browse_iPhone_T22_Click \x93Cruise&Stay\"\" link from cruise menu\"",
"SampleCount": "1394",
"column3": "101",
"column4": "92\r"
}
},
{
"_index": "pattwo",
"_type": "jmetern",
"_id": "S2_CR_Browse_desktop_T2_Destinations View All",
"_score": 1,
"_source": {
"message": "S2_CR_Browse_desktop_T2_Destinations View All,2785,354,288\r",
"@version": "1",
"@timestamp": "2016-07-19T17:19:11.986Z",
"path": "D:/pat20.csv",
"host": "BG3WS0057",
"type": "jmetern",
"label": "S2_CR_Browse_desktop_T2_Destinations View All",
"SampleCount": "2785",
"column3": "354",
"column4": "288"
}
}
]
}

2

Having both of these means your template won't be applied.[quote="AnantPatil, post:1, topic:55908"]
Here label field is having unique value so I am referring that as document id.
[/quote]

That's not really a good ID, but it should work.
Have you tried updating a document manually and seeing if the _version increments?

3

Hi warkolm,

Thank you for your reply.

Now I changed the template_overwrite from false to true in order to apply my customized template.
template => "D:/apache-jmeter-2.13/bin/jmeter-mapping.json"
template_overwrite => true

Considering the second point,label names are unique so passing document id as label name,remaining columns value will be changed every time so thought to mention the label names are document_id,Please let me know if you have any better idea so its great help to me.

The indexing getting updating if I changed the file name with different name then those new contents getting updated in existing index.

for Ex: a.csv is having two rows data and indexed
S2_CR_Browse_iPhone_T22_Click “Cruise&Stay" link from cruise menu,1394,101,92
S2_CR_Browse_desktop_T2_Destinations View All,2785,354,288

Now the content of a.csv got changed and updating these content in ealier generated index is not happening,Its happening only when I changed the a.csv to some other name like b.csv then these updated content are getting updated in existing index.any solution without changing the file name need to updated in existing index.

This is first time I am working on ELK.

4

Ahh ok, then that is likely a sincedb problem. Have a read of https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files