What is the best way to delete in elastic search and reparse an apache log file

path is available in ES records as one field(from where the record is got)

Deleting from Elastic search -

Currently I am doing this way -

curl -s -X POST http://localhost:9200/test-log-*/_delete_by_query?q=path:log1234

Reparse log file -

I have a logstash process running all the time, so

remove and copy the file in logstash path-

cd <logstash_path_dir>;rm log1234; cp <from_source> log1234

This works when I do delete and wait until it becomes zero records in ES and do reparse manually (only some times).
When ran as a script its not working ... Do I need to add more wait (sleep) in script for ES to flush the data?

Hi elk_dni1,

What versions are you using ? How is configured your logstash input plugin ? What OS are you running logstash on ?

Best regards,

Charles Casadei.

Linux OS, ES

Input -

input {
file {
type => "abc"
path => "/app/xyz/es-data/app-logs-6/abc/*access*"
codec => plain { charset => "ISO-8859-1" }
exclude => "*.gz"
sincedb_path => "/app/xyz/es-data/metadata-6/sincedb/sincedb_qwert"
max_open_files => 10000
ignore_older => 0

To make it new inode, I am removing file and copying again, but still logstash is not picking...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.