Marvel not collecting system metrics

Elastic Stack Elasticsearch
1

Hi,

I've been deploying Elasticsearch with Docker and it's been working great.
So I added the Marvel plugin and I get a ton of these in my Elasticsearch
error log and no system metrics in Marvel. :frowning:

[2014-02-14 02:55:02,753][DEBUG][action.search.type ] [Kamal]
[.marvel-2014.02.14][0], node[_Ofz6YYlTc2hA9ASu3qlRg], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@53b9f5]
org.elasticsearch.search.SearchParseException: [.marvel-2014.02.14][0]:
query[ConstantScore(BooleanFilter(+no_cache(@timestamp:[1392342894189 TO
1392346502752]) +cache(_type:node_stats) +cache(@timestamp:[1392345900000
TO 1392346560000])))],from[-1],size[0]: Parse Failure [Failed to parse
source
[{"size":0,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1392342894189,"to":"now"}}},{"term":{"_type":"node_stats"}},{"range":{"@timestamp":{"from":"now-10m/m","to":"now/m"}}}]}}}},"facets":{"timestamp":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"@timestamp","order":"term","size":2000}},"os.cpu.usage":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"os.cpu.usage","order":"term","size":2000}},"os.load_average.1m":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"os.load_average.1m","order":"term","size":2000}},"jvm.mem.heap_used_percent":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"jvm.mem.heap_used_percent","order":"term","size":2000}},"fs.total.available_in_bytes":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"fs.total.available_in_bytes","order":"term","size":2000}},"fs.total.disk_io_op":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"fs.total.disk_io_op","order":"term","size":2000}}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:586)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:489)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:474)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:467)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:239)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
at
org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:70)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [fs.total.available_in_bytes]: failed to find mapping for
fs.total.available_in_bytes
at
org.elasticsearch.search.facet.termsstats.TermsStatsFacetParser.parse(TermsStatsFacetParser.java:132)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:574)
... 12 more

Making Marvel only semi-useful; I don't have anything disabled in my
elasticsearch.yml it's pretty stock; any help?

I tried asking in IRC but #elasticsearch isn't very helpful with questions
in regards to Marvel.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/326afeb5-1282-4ccd-979e-689556f84403%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I forgot to mention, the Dockerfiles I use to build my Elasticsearch are at
GitHub - damm/dockerfiles: My Dockerfiles; feel free to review the java/ image for how I
build java and elasticsearch/ for ES.

Is it possible there's some external tools I need installed?

Thanks!

On Thu, Feb 13, 2014 at 6:57 PM, Scott Likens smlikens@gmail.com wrote:

Hi,

I've been deploying Elasticsearch with Docker and it's been working great.
So I added the Marvel plugin and I get a ton of these in my Elasticsearch
error log and no system metrics in Marvel. :frowning:

[2014-02-14 02:55:02,753][DEBUG][action.search.type ] [Kamal]
[.marvel-2014.02.14][0], node[_Ofz6YYlTc2hA9ASu3qlRg], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@53b9f5]
org.elasticsearch.search.SearchParseException: [.marvel-2014.02.14][0]:
query[ConstantScore(BooleanFilter(+no_cache(@timestamp:[1392342894189 TO
1392346502752]) +cache(_type:node_stats) +cache(@timestamp:[1392345900000
TO 1392346560000])))],from[-1],size[0]: Parse Failure [Failed to parse
source
[{"size":0,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1392342894189,"to":"now"}}},{"term":{"_type":"node_stats"}},{"range":{"@timestamp":{"from":"now-10m/m","to":"now/m"}}}]}}}},"facets":{"timestamp":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"@timestamp","order":"term","size":2000}},"os.cpu.usage":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"os.cpu.usage","order":"term","size":2000}},"os.load_average.1m":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"os.load_average.1m","order":"term","size":2000}},"jvm.mem.heap_used_percent":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"jvm.mem.heap_used_percent","order":"term","size":2000}},"fs.total.available_in_bytes":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"fs.total.available_in_bytes","order":"term","size":2000}},"fs.total.disk_io_op":{"terms_stats":{"key_field":"node.ip_port.raw","value_field":"fs.total.disk_io_op","order":"term","size":2000}}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:586)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:489)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:474)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:467)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:239)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
at
org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:70)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [fs.total.available_in_bytes]: failed to find mapping for
fs.total.available_in_bytes
at
org.elasticsearch.search.facet.termsstats.TermsStatsFacetParser.parse(TermsStatsFacetParser.java:132)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:574)
... 12 more

Making Marvel only semi-useful; I don't have anything disabled in my
elasticsearch.yml it's pretty stock; any help?

I tried asking in IRC but elasticsearch isn't very helpful with questions
in regards to Marvel.

Thanks!

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/326afeb5-1282-4ccd-979e-689556f84403%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4VnrbgMfGmcz_hKsnAo5re0Jun%2BttF101OBJ7vWVe6Btg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

How did you install Marvel?

You need to add Marvel plugin on every node and each node must be restarted.

Also take care of the setting marvel.agent.exporter.es.hosts if you use
other host/port than localhost:9200

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWrMoZPm8bHfkQbq_-uJojAFLQgaexu0UXXjz5M%2B0biQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi,

As I am using Docker Elasticsearch is not running during the build process.
I have posted my Dockerfile below.

FWIW my /_node/stats is correct and fully populated.

Additionally I have looked over
https://groups.google.com/forum/#!msg/elasticsearch/UPBMKvxwwM8/VBaqh0B11tQJand
tried applying that as a custom template and deleting my indices and
starting over; did not help.

Additionally the elasticsearch.yml in that repo is the current one in use.

Thanks!

On Fri, Feb 14, 2014 at 12:17 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

How did you install Marvel?

You need to add Marvel plugin on every node and each node must be
restarted.

Also take care of the setting marvel.agent.exporter.es.hosts if you use
other host/port than localhost:9200

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWrMoZPm8bHfkQbq_-uJojAFLQgaexu0UXXjz5M%2B0biQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4XJRFyWWk7mhHcKt2i%3D9-kXf8Ji54LQTLU4ebmAjTZNRA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

May I ask which OS you are running ES on?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c840c19d-e04d-4f51-bd3f-4c8bc7d7fe0d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hi scott

next to Binh's question can you check you have the following field in your
nodes stats( GET _node/stats): fs.total.available_in_bytes ?

On Friday, February 14, 2014 1:06:06 PM UTC+1, Binh Ly wrote:

May I ask which OS you are running ES on?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/466b1bd5-173d-46df-a8e2-39d52f2b8146%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

On Friday, February 14, 2014 4:06:06 AM UTC-8, Binh Ly wrote:

May I ask which OS you are running ES on?

IIRC Docker is a management tool for LXC.
So, it does pique the thoughts...

Where should marvel be pulling stats in an LXC deployment? It's not fully
isolated like other virtualization technologies and access to various Host
processes are blocked. I suppose it might be possible to point to
statistics derived by Docker, but that probably requires special
configuration.

Some thoughts,
Tony

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b409c6f0-1ea3-44f1-bffc-423edc4bb792%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Hi Boaz,

  "fs": {
    "data": [
      {
        "path": "/data/elasticsearch/shared/docker/nodes/0"
      }
    ],

Does not appear to be. This field is populated when I run it without
Docker; so is it expecting any particular file to exist like /etc/fstab or
/etc/mtab?

Thanks!

On Fri, Feb 14, 2014 at 7:35 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi scott

next to Binh's question can you check you have the following field in your
nodes stats( GET _node/stats): fs.total.available_in_bytes ?

On Friday, February 14, 2014 1:06:06 PM UTC+1, Binh Ly wrote:

May I ask which OS you are running ES on?

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/466b1bd5-173d-46df-a8e2-39d52f2b8146%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4U%3D__oqEDGVQQcUGK%2Ba9vvKMu8deJEEtrRRyDO7zW0BCw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Hi Binh,

My Java image is built off of the Docker image ubuntu:latest (which I
believe is 12.10) (Docker)

You are more than welcome to build the Docker image yourself from what I
have provided and reproduce this yourself.

Thanks!

On Fri, Feb 14, 2014 at 4:06 AM, Binh Ly binh@hibalo.com wrote:

May I ask which OS you are running ES on?

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c840c19d-e04d-4f51-bd3f-4c8bc7d7fe0d%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4W6%3DAB42HtV3NN8pZY-ew%3DLnHfwipA_2Bn1BLQ%3DihFWBg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

It does indeed sound like some metrics are not available from your
environment. Marvel/ES uses Sigar to collect these metrics
(https://support.hyperic.com/display/SIGAR/Home). Each OS has different
ways to provide (or not provide) these metrics. If you absolutely cannot
get these metrics, you can create a Marvel index template
(http://www.elasticsearch.org/guide/en/marvel/current/) and introduce these
missing fields that Sigar cannot collect from your environment.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/886a1c50-4eb6-4a87-99a2-d2916032e04d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Hi,

It looks like you depend on /etc/mtab or /etc/fstab. After adding some
simple ways to re-create the files in the container the "fs" key was
populated right. Unfortunately that Sigar page was not too helpful in
this; so I'm glad I was able to fix it myself.

I can't imagine I will be the first (nor the last) to hit simple stumbling
blocks like this; so I think it would be good to detail what is required
for Marvel to run correctly? Especially as the goal is to get a license fee
from us.

Thanks!

On Fri, Feb 14, 2014 at 12:09 PM, Binh Ly binh@hibalo.com wrote:

It does indeed sound like some metrics are not available from your
environment. Marvel/ES uses Sigar to collect these metrics (
https://support.hyperic.com/display/SIGAR/Home). Each OS has different
ways to provide (or not provide) these metrics. If you absolutely cannot
get these metrics, you can create a Marvel index template (
Elasticsearch Platform — Find real-time answers at scale | Elastic) and introduce
these missing fields that Sigar cannot collect from your environment.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/886a1c50-4eb6-4a87-99a2-d2916032e04d%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4XnSZFYmwbYehEDZUmWCSUq%2BopcxVm%2BDd4cJ2pCkwzbRQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

Very cool Scott, thanks for the information!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17d56270-1b96-4105-b1ee-754071655c05%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

Tony,

We appear to be getting the information back in the cgroup currently; so
what's provided is fairly good. It would be better obviously to grab them
from the cgroup and push it in; but that would be external to the container
(unless you mounted the cgroups in the container)

So for my case it works; some spots I do get the 'host' data and not the
container data but it's good enough for the moment.

Scott

On Fri, Feb 14, 2014 at 7:41 AM, Tony Su tonysu999@gmail.com wrote:

On Friday, February 14, 2014 4:06:06 AM UTC-8, Binh Ly wrote:

May I ask which OS you are running ES on?

IIRC Docker is a management tool for LXC.
So, it does pique the thoughts...

Where should marvel be pulling stats in an LXC deployment? It's not fully
isolated like other virtualization technologies and access to various Host
processes are blocked. I suppose it might be possible to point to
statistics derived by Docker, but that probably requires special
configuration.

Some thoughts,
Tony

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/dhpxaOuoZWI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b409c6f0-1ea3-44f1-bffc-423edc4bb792%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALDJu4XXVEJSdb4zs9sF6%3DgM92UK15K7A%2BUJ%2B8ugBg6Kn_yB_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.