Parsing variable to Curl delete command

Elastic Stack Elasticsearch
1

HI Team,
Actually im work in autopurge script in linux. I dont have enough storage in my disk. i planned to purge the old data from my index and keep last 15 day live records in index.. I can perform this operation by manually like passing this code as shown below,

curl -XDELETE 'http://XXX/logstash-dd.notifychange_log_v1/_query' -d '
{
"query":
{
"range":
{
"eventType_timestamp":
{
"gte": "2016-05-30T07:00:00.000Z",
"lte": "2016-06-15T06:59:59.999Z"
}
}
}
}'

In automation im trying to pass the from and to date automatically by script . I stored the from and to date value in variable and then passing the variable to curl command as shown below,
from_dataset_date=2016-05-30T07:00:00.000Z
to_dataset_date=2016-06-15T06:59:59.999Z
curl -XDELETE 'http://XXXX/logstash-dd.notifychange_log_v1/_query' -d '
{
"query":
{
"range":
{
"eventType_timestamp":
{
"gte": "$from_dataset_date",
"lte": "$to_dataset_date"

  }
}

}
}'

In this scenario im getting error.
error :{"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [$from_dataset_date] with format [strict_date_optional_time||epoch_millis]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-dd.notifychange_log_v1","node":"ze7WyKOcRRu0e5iAgA0ueQ","reason":{"type":"parse_exception","reason":"failed to parse date field [$from_dataset_date] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "$from_dataset_date""}}}]},"status":400}

Please anyone help me to resolve is it possible to pass variable in curl command as i shown above?

2

Variable references aren't expanded inside single quotes. They will be expanded inside double quotes, but then you need to make sure to escape the double quotes in the JSON string.

Deleting data like this is a bad idea. You should use time-series indexes instead and use Curator to drop indexes older than a certain age.

3

This X 100000000000000.

4

You are much better off inserting documents into an index named with the
current date and deleting the index when you don't need it any more. You
can use an alias to query them all. That way the delete frees up space.

5

Hi Magnusbaeck/warkolm,
In kibana while creating index with time-series its show time series will be deprecated . so that only we just go up with this method.

6

No, the concept of time-series indexes will not be deprecated. What's deprecated is Kibana's use of date patterns in the index names, i.e. "prefix-*" works just as well as an index pattern as "[prefix-]YYYY.MM.DD".

7

Thank you so much for your reply magnus :slight_smile:

8

I will go up with your point before that i want to know is it possible to pass the value stored variable in curl query as i mentioned above code? Can you explain that to me.

9

Did you read my previous response? Note that this is a general Unix shell quoting problem, unrelated to Elasticsearch.

10

Ok got it.. Thank you for your response.