Issue with facets searching multiple indices when field is missing

Hi,

I am kibana (3.1) & elasticsearch (1.3.2).

We collect nginx and tomcat logs from multiple machines with logstash and
display this in kibana, so far so good.

We are now trying to correlate this with system resources, so we use
collectd + logstash to store collectd info.* This is stored in a separate
index*.

We want to use this information in the same view as our logs so we want to
search multiple indices
"[logstash-nginx-frontend-]YYYY.MM.DD,[logstash-collectd-]YYYY.MM.DD".

However, since other indices does not contain this field, the query fails
(see stacktrace below). If we only search over our collectd indice it works
flawlessly.

Is this expected behaviour? If it is, is there anyway we can get by this?

Kind regards /Johan

// This is with topn query but we get the same result with lucene query

org.elasticsearch.search.SearchParseException:
[logstash-tomcat-frontend-2014.10.02][0]: from[-1],size[-1]: Parse Failure
[Failed to parse source
[{"facets":{"43":{"date_histogram":{"key_field":"@timestamp","value_field":"value","interval":"5m"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"host:"frontend-dev6"
AND (plugin:df AND
collectd_type:percent_bytes)"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1412201724761,"to":1412244924761}}}]}}}}}}},"44":{"date_histogram":{"key_field":"@timestamp","value_field":"value","interval":"5m"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"host:"frontend-dev6"
AND (plugin:df AND
collectd_type:percent_bytes)"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1412201724761,"to":1412244924761}}}]}}}}}}}},"size":0}]]

    at 

org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)

    at 

org.elasticsearch.search.SearchService.createContext(SearchService.java:515)

    at 

org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)

    at 

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:256)

    at 

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)

    at 

org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)

    at 

org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)

    at 

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at 

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [43]: (value) field [value] not found

    at 

org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:195)

    at 

org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)

    at 

org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa0c73ed-a0ba-4667-ab28-23673a88d94d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Does anyone have a solution to this? I'm facing the same problem as well.
Not every index will have specific fields, but I would like to plot the
histograph over a period of time showing indices if the field does indeed
appear.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d61c743-fd9a-40b6-8bfd-c4b4eee7cfb8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

The only solution I could find was to set these fields in other indices as
well, like this.

if "collectd" not in [tags] {
mutate {
add_field => {
"tx" => "0"
"rx" => "0"
"shortterm" => "0"
"longterm" => "0"
"midterm" => "0"
"value" => "0"
}
}
}

Den måndagen den 3:e november 2014 kl. 14:32:04 UTC+1 skrev Erbin Lim:

Does anyone have a solution to this? I'm facing the same problem as well.
Not every index will have specific fields, but I would like to plot the
histograph over a period of time showing indices if the field does indeed
appear.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/139d1d02-a1c1-4fb0-942a-cabce7678e4b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Awesome thanks. Seems to work.

On Monday, November 3, 2014 2:36:14 PM UTC+1, Johan Rask wrote:

Hi,

The only solution I could find was to set these fields in other indices as
well, like this.

if "collectd" not in [tags] {
mutate {
add_field => {
"tx" => "0"
"rx" => "0"
"shortterm" => "0"
"longterm" => "0"
"midterm" => "0"
"value" => "0"
}
}
}

Den måndagen den 3:e november 2014 kl. 14:32:04 UTC+1 skrev Erbin Lim:

Does anyone have a solution to this? I'm facing the same problem as well.
Not every index will have specific fields, but I would like to plot the
histograph over a period of time showing indices if the field does indeed
appear.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/825fcced-8d83-4b2a-9043-5e00b9c3fc38%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.