Help deleting data

keithh · February 28, 2018, 12:31pm

Hi there,

Forgive my ignorance in advance.

I've inherited an old stack with elasticsearch 1.7. Previously, as has been documented in many threads, it was possible to delete logs using a chrome plugin, which has since been disabled.

I'm reluctant to upgrade the stack because I don't know what else I'll need to do and I have no internal resources to draw on. Previously I would delete logstash logs older than a certain date just by changing the date parameter in a query in the plugin console on Chrome. Please could someone give me some help in how I delete logs older than a specific date going forward?

I can't remember the exact syntax but it went something like "DELETE LOGSTASH ...."

Thanks in advance (and apologies once again if this is a dumb question).
Keith

paz · February 28, 2018, 1:24pm

This seems more appropriate for the ElasticSeach subforum

Anyhow, delete-by-query is probably what you're looking for, where you can query for all documents older than a specific date and delete them.

keithh · February 28, 2018, 1:37pm

Thanks @paz,

Apologies for mis-posting. I take it that delete-by-query can only be invoked from the command line?

paz · February 28, 2018, 1:39pm

I'd assume you can also run it via the head plugin, though I have never tested so myself.

keithh · March 8, 2018, 12:19pm

Thanks @paz ,

Apologies once more for my ignorance. what plugin are you referring to? Is it a Chrome plugin?

paz · March 8, 2018, 12:34pm

No worries. It's this one. Depending on your ElasticSearch version you can either install it as an ElasticSearch plugin or a standalone version.

If with "Chrome plugin" you mean this, it's practically the same thing as above wrapped in a browser extension, so you can use that indeed.

keithh · April 4, 2018, 10:58am

Thanks @paz,

Sorted now. Installed the plugin. It took me a bit to get my head round it. I couldn't find any docs for the chrome plugin, so was trying all sorts of complicated structured queries, when I stumbled upon the delete option in the Actions dropdown. D'Oh!

Happy now!

theuntergeek · April 4, 2018, 3:03pm

Oh wow. Delete by query is a horrible idea for time series data. It's like the difference in SQL between DELETE from TABLE where timestamp < x and DROP TABLE. Millions of atomic operations in one vs. a single operation for the other.

Use Elasticsearch Curator to delete time series indices on a schedule.

system · May 2, 2018, 3:03pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.