Aggs / Max timestamp Term query issue

[mgolubov55 (Michael Golubov)]

When I execute this query:
GET /logstash*/_search?size=0
{
"aggs" : {
"max_timestamp" : { "max" : { "field" : "@timestamp" } }
}
}
I get this result:
.
.
},
"aggregations" : {
"max_timestamp" : {
"value" : 1.544640671569E12,
"value_as_string" : "2018-12-12T18:51:11.569Z"
}
}
}

However, when I execute the following query:
GET /logstash*/_search
{
"aggs" : {
"max_timestamp" : { "max" : { "field" : "@timestamp" } }
},
"query": {
"term": {"@timestamp": "max_timestamp"}
}
}
I get the following results:
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "failed to parse date field [max_timestamp] with format [strict_date_optional_time||epoch_millis]"
},
{
"type": "parse_exception",
"reason": "failed to parse date field [max_timestamp] with format [strict_date_optional_time||epoch_millis]"
},
{
"type": "parse_exception",
"reason": "failed to parse date field [max_timestamp] with format [strict_date_optional_time||epoch_millis]"
},

How can I correct this situation?

[xavierfacq]

Hi,

First I suggest to say "hi or hello", "bye", "thanks". and give details about your ES version etc...

Read : https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html
Because this not a valid query :

"term": {"@timestamp": "max_timestamp"}

bye ,
Xavier

[mgolubov55 (Michael Golubov)]

Xavier,

Thank you for your reply.

I am attaching a Word doc that outlines the issue.

Regards,

Michael

Michael Golubov

QT IT - Trading Tools - US

()

(Attachment Xavier_ELK1.docx is missing)

[mgolubov55 (Michael Golubov)]

Xavier,

Thank you for replying – here is what I am attempting to do

1.  Here is an aggs / max search where I have copied the value of the return
   

2.  Manually inserting the value into a term search, I get exactly what I need:
   

3.   Given the above two search, I was hoping that “max_timestamp” would be treated as a dynamic variable which I could use as follows below. Clearly, it doesn’t work that way, and I am hoping that there is some way I can ‘pipeline’ the result of an aggs / max search into a term search. I am using version 6.5.0 of ELK.
   

Thank you,

Michael

Michael Golubov

QT IT - Trading Tools - US

()

[xavierfacq]

Hi,

I'm not certain of what you are expected, but you have to know that in the query ("term": {"@timestamp": "max_timestamp"}) you have to provide a value that will be used to filter documents.
Relation between query and aggregration are not in the way you are expected.

You run a query on your documents, they are filtered with query elements (timestamp, field value etc..) and finaly you can have aggregations based on this results.

Hope it's clear,
Xavier

[xavierfacq]

Just a question: what are you looking for into your documents ?

[mgolubov55 (Michael Golubov)]

Xavier,

Replying to your latest post:

I need to monitor a discrete set of services (11 in this case) sent by metricbeat via logstash into elasticsearch.

I need to make sure that only 11 are displayed when the data is refreshed from logstash (every 30 seconds) The following query works most of the time:

However, there are instances where the following happens. Duplicated data (11 extra) is displayed for 5 seconds and is cleared on refresh. I am trying to adjust
the above query to only display data from the latest timestamp in the range. I think that the query examples I sent would work to do this if the latest timestamp could be calculated in the term query.

Michael Golubov

QT IT - Trading Tools - US

()

[xavierfacq]

Ok if I resume, you want to see the lastest update of each service , is that ok ?

[mgolubov55 (Michael Golubov)]

Yes, the display would only have to be updated if the status of a service changes.