'Shard Allocation' dashboard in Marvel 1.2 showing SearchParseExceptions

philsttr · June 6, 2014, 10:51pm

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch], all
shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException: [.marvel-2014.06.06][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1aeb0c36-9cd5-4f2b-a8c7-9b32471b4716%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boaz_Leskes · June 7, 2014, 6:50pm

Hi,

Is there anything in the log indicating data shipping problems from the
agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch], all
shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException: [.marvel-2014.06.06][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e26069b-c8e3-4963-bec3-2d524945e2e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

philsttr · June 9, 2014, 4:35pm

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx] exporter
[es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from the
agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch], all
shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException: [.marvel-2014.06.06][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/820ec77c-0d23-42f7-b966-737e6304085f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

philsttr · June 9, 2014, 4:39pm

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from the
agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch], all
shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException: [.marvel-2014.06.06][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2393de32-1eef-4fe3-868d-a1fb72f948be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boaz_Leskes · June 9, 2014, 5:01pm

Yeah, I suspected that. Let me know how it goes...

On Mon, Jun 9, 2014 at 6:39 PM, null philsttr@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.
Elasticsearch Platform — Find real-time answers at scale | Elastic
On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from the
agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch], all
shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException: [.marvel-2014.06.06][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/YET1SifrKy4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2393de32-1eef-4fe3-868d-a1fb72f948be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1402333315298.de2a2bbe%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.

return_null · September 17, 2014, 4:52pm

I am on Marvel 1.2.1 and had the same error (see below). The time on one of
my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the Troll]
All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the Troll]
[.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException: [.marvel-2014.09.17][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from the
agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch],
all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException: [.marvel-2014.06.06][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/33caa09e-3db5-4efc-94e5-b258c0ecca96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

rahst12 · November 7, 2014, 4:54pm

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on one
of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException: [.marvel-2014.09.17][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from the
agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see the
following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch],
all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx] All
shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f83d9fe9-e4d4-4e66-8dba-5386e50d3512%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Max_Charas · November 25, 2014, 9:29am

Same problem here.

Log says:

Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
Verified that the index(es) have the timestamp field.

Running Marvel 1.2.1 and Elasticsearch 1.3.2.

Any help would be appreciated as we cant get our Marvel up and running.

Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
Hendrickson:

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on one
of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException: [.marvel-2014.09.17][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from
the agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see
the following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch],
all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
All shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ddb93440-ea7f-4a73-a632-eb4615ca458a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Teo_Ruiz · November 26, 2014, 4:24pm

Hi guys,

We're seeing exactly the same error. This is the error we see:
org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort":{"
@timestamp":{"order":"desc"}}}]]

Full trace
here: marvel failure with mapping · GitHub
Mappings
here: marvel failure with mapping · GitHub

ES version on monitoring and production clusters: 1.3.4
Marvel version on monitoring and production clusters: 1.2.1

I'm not sure how to proceed, since we removed and reinstalled marvel
already. The problem started showing when we upgraded the production
cluster from 0.90.x to 1.3.4.

I'd appreciate some light shedding on this

On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:

Same problem here.

Log says:

Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
Verified that the index(es) have the timestamp field.

Running Marvel 1.2.1 and Elasticsearch 1.3.2.

Any help would be appreciated as we cant get our Marvel up and running.

Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
Hendrickson:

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on one
of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException: [.marvel-2014.09.17][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from
the agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see
the following error:

SearchPhaseExecutionException[Failed to execute phase [query_fetch],
all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
All shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the
cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e897ccba-e5ac-49c6-baa3-6f43208bf095%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boaz_Leskes · November 30, 2014, 10:45pm

Hi Teo, Max,

From the query, I can see that it that cluster_state is not shipped. The
marvel index template adds the cluster_type with some non-default fields
but all fields (like @timestamp) are added when the first document is
shipped.

Can you validate there are no documents of that type?

Further a couple of clarifying questions:

Did you install the marvel agent on all cluster nodes? i.e., also master
nodes? The later is the ones that ship that data..
Did you upgrade from a previous version of marvel or is it a fresh
install?

Cheers,
Boaz

On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:

Hi guys,

We're seeing exactly the same error. This is the error we see:
org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort":
{"@timestamp":{"order":"desc"}}}]]

Full trace here:
marvel failure with mapping · GitHub
Mappings here:
marvel failure with mapping · GitHub

ES version on monitoring and production clusters: 1.3.4
Marvel version on monitoring and production clusters: 1.2.1

I'm not sure how to proceed, since we removed and reinstalled marvel
already. The problem started showing when we upgraded the production
cluster from 0.90.x to 1.3.4.

I'd appreciate some light shedding on this

On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:

Same problem here.

Log says:

Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
Verified that the index(es) have the timestamp field.

Running Marvel 1.2.1 and Elasticsearch 1.3.2.

Any help would be appreciated as we cant get our Marvel up and running.

Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
Hendrickson:

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on
one of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException: [.marvel-2014.09.17][0]:
from[-1],size[1]: Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from
the agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com
wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see
the following error:

SearchPhaseExecutionException[Failed to execute phase
[query_fetch], all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
All shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the
cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0519c6d-0cba-4f55-9521-0ce6c945f284%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Max_Charas · December 1, 2014, 7:17pm

Hello Boaz,
Thanks for helping!

I read through your post a few times and realized that it was probably the
master node that wasn't sending any date as you indicated. A restart of the
node fixed the problem.

I have no idea why it wasn't shipping any data to start with, most likely
we hadn't restarted it since we installed the plugin but I'm not completely
sure.

In any case its working now.

Many thanks,
Max

Den söndagen den 30:e november 2014 kl. 23:45:15 UTC+1 skrev Boaz Leskes:

Hi Teo, Max,

From the query, I can see that it that cluster_state is not shipped. The
marvel index template adds the cluster_type with some non-default fields
but all fields (like @timestamp) are added when the first document is
shipped.

Can you validate there are no documents of that type?

Further a couple of clarifying questions:

Did you install the marvel agent on all cluster nodes? i.e., also
master nodes? The later is the ones that ship that data..

Did you upgrade from a previous version of marvel or is it a fresh
install?

Cheers,
Boaz

On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:

Hi guys,

We're seeing exactly the same error. This is the error we see:
org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort"
:{"@timestamp":{"order":"desc"}}}]]

Full trace here:
marvel failure with mapping · GitHub
Mappings here:
marvel failure with mapping · GitHub

ES version on monitoring and production clusters: 1.3.4
Marvel version on monitoring and production clusters: 1.2.1

I'm not sure how to proceed, since we removed and reinstalled marvel
already. The problem started showing when we upgraded the production
cluster from 0.90.x to 1.3.4.

I'd appreciate some light shedding on this

On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:

Same problem here.

Log says:

Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
Verified that the index(es) have the timestamp field.

Running Marvel 1.2.1 and Elasticsearch 1.3.2.

Any help would be appreciated as we cant get our Marvel up and running.

Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
Hendrickson:

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on
one of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from
the agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com
wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see
the following error:

SearchPhaseExecutionException[Failed to execute phase
[query_fetch], all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
All shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the
cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa6036bc-1979-4b0c-a7f2-e35c4a96d80f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Teo_Ruiz · December 2, 2014, 10:07am

Hi Boaz,

I finally restarted the master node and it worked, indeed. It's anyway a
quite confusing error message / situation

Cheers,

On Sunday, November 30, 2014 11:45:15 PM UTC+1, Boaz Leskes wrote:

Hi Teo, Max,

From the query, I can see that it that cluster_state is not shipped. The
marvel index template adds the cluster_type with some non-default fields
but all fields (like @timestamp) are added when the first document is
shipped.

Can you validate there are no documents of that type?

Further a couple of clarifying questions:

Did you install the marvel agent on all cluster nodes? i.e., also
master nodes? The later is the ones that ship that data..

Did you upgrade from a previous version of marvel or is it a fresh
install?

Cheers,
Boaz

On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:

Hi guys,

We're seeing exactly the same error. This is the error we see:
org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort"
:{"@timestamp":{"order":"desc"}}}]]

Full trace here:
marvel failure with mapping · GitHub
Mappings here:
marvel failure with mapping · GitHub

ES version on monitoring and production clusters: 1.3.4
Marvel version on monitoring and production clusters: 1.2.1

I'm not sure how to proceed, since we removed and reinstalled marvel
already. The problem started showing when we upgraded the production
cluster from 0.90.x to 1.3.4.

I'd appreciate some light shedding on this

On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:

Same problem here.

Log says:

Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
Verified that the index(es) have the timestamp field.

Running Marvel 1.2.1 and Elasticsearch 1.3.2.

Any help would be appreciated as we cant get our Marvel up and running.

Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
Hendrickson:

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on
one of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from
the agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com
wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see
the following error:

SearchPhaseExecutionException[Failed to execute phase
[query_fetch], all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
All shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the
cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/14c54640-a555-4b6e-be4a-5a52d1fb2f7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boaz_Leskes · December 2, 2014, 10:46am

Agreed. Happy things work now.

On Tue, Dec 2, 2014 at 11:07 AM, Teo Ruiz teoruiz@gmail.com wrote:

Hi Boaz,
I finally restarted the master node and it worked, indeed. It's anyway a
quite confusing error message / situation
Cheers,
On Sunday, November 30, 2014 11:45:15 PM UTC+1, Boaz Leskes wrote:

Hi Teo, Max,

From the query, I can see that it that cluster_state is not shipped. The
marvel index template adds the cluster_type with some non-default fields
but all fields (like @timestamp) are added when the first document is
shipped.

Can you validate there are no documents of that type?

Further a couple of clarifying questions:

Did you install the marvel agent on all cluster nodes? i.e., also
master nodes? The later is the ones that ship that data..

Did you upgrade from a previous version of marvel or is it a fresh
install?

Cheers,
Boaz

On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:

Hi guys,

We're seeing exactly the same error. This is the error we see:
org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort"
:{"@timestamp":{"order":"desc"}}}]]

Full trace here:
marvel failure with mapping · GitHub
Mappings here:
marvel failure with mapping · GitHub

ES version on monitoring and production clusters: 1.3.4
Marvel version on monitoring and production clusters: 1.2.1

I'm not sure how to proceed, since we removed and reinstalled marvel
already. The problem started showing when we upgraded the production
cluster from 0.90.x to 1.3.4.

I'd appreciate some light shedding on this

On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:

Same problem here.

Log says:

Parse Failure [Failed to parse source
[{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
Verified that the index(es) have the timestamp field.

Running Marvel 1.2.1 and Elasticsearch 1.3.2.

Any help would be appreciated as we cant get our Marvel up and running.

Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
Hendrickson:

I see the same error Were you able to resolve it?

Thanks

On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:

I am on Marvel 1.2.1 and had the same error (see below). The time on
one of my nodes was off a fair bit. Fixed that and the problem went away.

[2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
Troll] All shards failed for phase: [query_fetch]
[2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@769694e9]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
... 9 more

On Monday, June 9, 2014 12:39:45 PM UTC-4, phil...@gmail.com wrote:

Looks like this has already been found and resolved. I'll upgrade to
Marvel 1.2.1.

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, June 9, 2014 9:35:30 AM UTC-7, phil...@gmail.com wrote:

Hi Boaz,

The only other exception I see in the logs is this:

[2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
exporter [es_exporter] has thrown an exception:
java.lang.IllegalStateException: array not available
at
org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
at java.lang.Thread.run(Thread.java:745)

On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:

Hi,

Is there anything in the log indicating data shipping problems from
the agent? Check the log of the current master node.

Cheers,
Boaz

On Saturday, June 7, 2014 12:51:44 AM UTC+2, phil...@gmail.com
wrote:

I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.

When I go to the new "Shard Allocation" dashboard in Marvel, I see
the following error:

SearchPhaseExecutionException[Failed to execute phase
[query_fetch], all shards failed; shardFailures
{[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [Failed to parse source
[{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
nested: SearchParseException[[.marvel-2014.06.04][0]: from[0],size[300]:
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]

I see the following in my elasticsearch.log

[2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
All shards failed for phase: [query_fetch]
[2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
[.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@6af079e8]
org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to parse
source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
at
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.SearchParseException:
[.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping found
for [@timestamp] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

I have confirmed that marvel is upgraded on every node in the
cluster.

I tried deleting the .marvel* indexes. Didn't help
Shutdown the cluster, and restarted it. Didn't help.

Anybody have ideas?

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/YET1SifrKy4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/14c54640-a555-4b6e-be4a-5a52d1fb2f7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1417517208059.f25f0b45%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.