XDELETE remove no records

Elastic Stack Elasticsearch
1

Hello

We use Elasticsearch Version is 1.7.1 and i have this script and want delete old messages on Graylog.

#!/bin/sh
FDATE=date +"%F %H:%M:%S.000" --date="60 days ago"
TDATE=date +"%F %H:%M:%S.000" --date="0 days ago"

RANGE='{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "level:7",
"allow_leading_wildcard": false
}
},
"filter": {
"bool": {
"must": {
"range": {
"timestamp": {
"from": "'${FDATE}'",
"to": "'${TDATE}'",
"include_lower": true,
"include_upper": true
}
}
}
}
}
}
}
}'
/usr/bin/curl -v3 -H "Content-Type: application/json" -XDELETE "http://127.0.0.1:9200/graylog2_*/message/_query" -d "${RANGE}"

When i run this script, then i have this output

  • About to connect() to 127.0.0.1 port 9200 (#0)
  • Trying 127.0.0.1... connected
  • Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)

DELETE /graylog2_*/message/_query HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: 127.0.0.1:9200
Accept: /
Content-Type: application/json
Content-Length: 523

< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 203
<

  • Connection #0 to host 127.0.0.1 left intact
  • Closing connection #0
    {"_indices":{"graylog2_3":{"_shards":{"total":4,"successful":4,"failed":0}},"graylog2_2":{"_shards":{"total":4,"successful":4,"failed":0}},"graylog2_4":{"_shards":{"total":4,"successful":4,"failed":0}}}}
    [blcrood@blx-lm00 scripts]$

No records will be deleted and i have no error. What is the reason because no records will be deleted?

regards
Steve

2

Try to use @timestamp field instead of timestamp .
Also for @timestamp you should use date format with 'T' :
for example:
date +"%FT%H:%M:%S.000" --date="60 days ago"

3

Doesn't graylog use time based indices?

4

Thank you, i can my script to

FDATE=date +"%FT%H:%M:%S.000" --date="60 days ago"
TDATE=date +"%FT%H:%M:%S.000" --date="0 days ago"

and when i then run the script, then i have this error.

{"_indices":{"graylog2_3":{"_shards":{"total":4,"successful":0,"failed":4,"failures":[{"index":"graylog2_3","shard":0,"reason":"QueryParsingException[[graylog2_3] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_3","shard":1,"reason":"QueryParsingException[[graylog2_3] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_3","shard":2,"reason":"QueryParsingException[[graylog2_3] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_3","shard":3,"reason":"QueryParsingException[[graylog2_3] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "}]}},"graylog2_2":{"_shards":{"total":4,"successful":0,"failed":4,"failures":[{"index":"graylog2_2","shard":0,"reason":"QueryParsingException[[graylog2_2] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_2","shard":1,"reason":"QueryParsingException[[graylog2_2] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_2","shard":2,"reason":"QueryParsingException[[graylog2_2] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_2","shard":3,"reason":"QueryParsingException[[graylog2_2] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "}]}},"graylog2_4":{"_shards":{"total":4,"successful":0,"failed":4,"failures":[{"index":"graylog2_4","shard":0,"reason":"QueryParsingException[[graylog2_4] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number]; nested: IllegalArgumentException[Invalid format: ""]; "},{"index":"graylog2_4","shard":1,"reason":"QueryParsingException[[graylog2_4] Failed to parse]; nested: ElasticsearchParseException[failed to parse date field [], tried both date format [yy* Connection #0 to host 127.0.0.1 left intact

Is my date format wrong?

regards
Steve

5

Hi,

This error means that the date format you are using in from/to in your Range query cannot be parsed by the one that is defined in your date field mapping (which looks like the default one). You should try to use the format parameter to match the date format that your script is inserting into the query. The format parameter supports another date format than the one defined in mapping.

Also, since the field name is not shown correctly in the error message (field []) you should check again how the field is called in your index. As @kirill_polishchuk mentioned, it could be @timestamp.

6

Thank you very much, but sorry i'm not so the format guru for a bash script. The original graylog query is like that.

"range": {
"timestamp": {
"from": "2016-02-29 05:48:29.700",
"to": "2016-03-30 05:48:29.700",
"include_lower": true,
"include_upper": true
}
}

What is the right format form my variables (FDATE and TDATE)?

regards
Steve

7

With me original code

$ date +"%F %H:%M:%S.000"
2016-03-30 11:24:57.000

i have same syntax like the original graylog query. I have no errors but no messages will be deleted. I don't understand when i use the same query with XGET (not XDELETE), then i have a output with

"hits" : {
"total" : 1300060,

it find messages to delete but when i use the same query with XDELETE not works.

regards
Steve