What is the best way to delete in elastic search and reparse an apache log file

[elk_dni_1 (elkuser_dni_1)]

path is available in ES records as one field(from where the record is got)

Deleting from Elastic search -

Currently I am doing this way -

curl -s -X POST http://localhost:9200/test-log-*/_delete_by_query?q=path:log1234

Reparse log file -

I have a logstash process running all the time, so

remove and copy the file in logstash path-

cd <logstash_path_dir>;rm log1234; cp <from_source> log1234

This works when I do delete and wait until it becomes zero records in ES and do reparse manually (only some times).
When ran as a script its not working ... Do I need to add more wait (sleep) in script for ES to flush the data?

[Charles.w (Charles Casadei)]

Hi elk_dni1,

What versions are you using ? How is configured your logstash input plugin ? What OS are you running logstash on ?

Best regards,

Charles Casadei.

[elk_dni_1 (elkuser_dni_1)]

Linux OS, ES 6.0.0.0

Input -

input {
file {
type => "abc"
path => "/app/xyz/es-data/app-logs-6/abc/*access*"
codec => plain { charset => "ISO-8859-1" }
exclude => "*.gz"
sincedb_path => "/app/xyz/es-data/metadata-6/sincedb/sincedb_qwert"
max_open_files => 10000
ignore_older => 0
}

To make it new inode, I am removing file and copying again, but still logstash is not picking...