Writing a ES plugin for AWS CloudWatch

Hi all,

I'm writing an elasticsearch plugin which periodically (e.g. every minute)
checks the health and state information of a cluster, and posts this
information in Amazon Web Service's CloudWatch. In this way we can monitor
what is going on with our database, and we are able to act as soon as some
of these values are out of the normal range.

I'm pretty much a newbie in elasticsearch, and I read somewhere in another
posthttps://groups.google.com/forum/?hl=en&fromgroups=#!searchin/elasticsearch/writing$20plugin/elasticsearch/fsTVZg3Hyr8/IQfmGI1jMqYJthat there are these kinds of plugins:

  • "river plugins" operating at cluster level
  • "generic plugins" working at node level
  • "index plugins" working at indices level, having access to mappings and
    indices within a cluster
  • "shard plugins" working at shard level, having access to Lucene index
    structures
  • "transport client plugins" working out-of-scope of a cluster

So what should be the kind of plugin I should use? I need to be able to:

  • Run periodically. I plan to use a java Timer for that.
  • Access information about the cluster state and cluster health.

Which other plugin can I look at, to get an example of how to do the second?

Thanks!!

Flavia

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi!

I made progress with the plugin, it's nearly done. I wrote it using a
service (inheriting AbstractLifecycleComponent), like this one:
https://github.com/elasticsearch/elasticsearch-transport-thrift. The
authors of this plugin also wrote a guide/intro to writing a plugin, see
here: http://jfarrell.github.com/, very nice.

Still it's not clear to me what kind of plugins you can write. It seems
every plugin I look at is implemented in a different way. Perhaps there is
some documentation about it that I didn't find.

Thanks!
Flavia

On Thursday, March 7, 2013 11:24:32 AM UTC+1, fla...@9apps.net wrote:

Hi all,

I'm writing an elasticsearch plugin which periodically (e.g. every minute)
checks the health and state information of a cluster, and posts this
information in Amazon Web Service's CloudWatch. In this way we can monitor
what is going on with our database, and we are able to act as soon as some
of these values are out of the normal range.

I'm pretty much a newbie in elasticsearch, and I read somewhere in another
posthttps://groups.google.com/forum/?hl=en&fromgroups=#!searchin/elasticsearch/writing$20plugin/elasticsearch/fsTVZg3Hyr8/IQfmGI1jMqYJthat there are these kinds of plugins:

  • "river plugins" operating at cluster level
  • "generic plugins" working at node level
  • "index plugins" working at indices level, having access to mappings and
    indices within a cluster
  • "shard plugins" working at shard level, having access to Lucene index
    structures
  • "transport client plugins" working out-of-scope of a cluster

So what should be the kind of plugin I should use? I need to be able to:

  • Run periodically. I plan to use a java Timer for that.
  • Access information about the cluster state and cluster health.

Which other plugin can I look at, to get an example of how to do the
second?

Thanks!!

Flavia

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You already have found out about plugin types. Because ES is so
flexible, you can write plugins in many different ways, but there are
some method usages plugins share.

In the class org.elasticsearch.plugins.AbstractPlugin you can see
methods a plugin can activate. The simplest type is the "generic"
plugin, I call it "generic" just because it does not care about indices,
shards, or transport operations, so, it does not use
indexModules/indexServices or shardModules/shardServices methods.

For your task of reading the cluster state and post it somewhere, a
generic plugin is ok.

Jörg

Am 08.03.13 12:47, schrieb flavia@9apps.net:

Still it's not clear to me what kind of plugins you can write. It
seems every plugin I look at is implemented in a different way.
Perhaps there is some documentation about it that I didn't find.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey there,

take a look at https://github.com/spinscale/elasticsearch-graphite-plugin

This plugin already does something similar you seem to try. It is polling
some cluster information every n seconds and pushes them somewhere. It
seems you are trying to do the same, so you might copy some code off it.

--Alex

On Fri, Mar 8, 2013 at 12:47 PM, flavia@9apps.net wrote:

Hi!

I made progress with the plugin, it's nearly done. I wrote it using a
service (inheriting AbstractLifecycleComponent), like this one:
https://github.com/elasticsearch/elasticsearch-transport-thrift. The
authors of this plugin also wrote a guide/intro to writing a plugin, see
here: http://jfarrell.github.com/, very nice.

Still it's not clear to me what kind of plugins you can write. It seems
every plugin I look at is implemented in a different way. Perhaps there is
some documentation about it that I didn't find.

Thanks!
Flavia

On Thursday, March 7, 2013 11:24:32 AM UTC+1, fla...@9apps.net wrote:

Hi all,

I'm writing an elasticsearch plugin which periodically (e.g. every
minute) checks the health and state information of a cluster, and posts
this information in Amazon Web Service's CloudWatch. In this way we can
monitor what is going on with our database, and we are able to act as soon
as some of these values are out of the normal range.

I'm pretty much a newbie in elasticsearch, and I read somewhere in another
posthttps://groups.google.com/forum/?hl=en&fromgroups=#!searchin/elasticsearch/writing$20plugin/elasticsearch/fsTVZg3Hyr8/IQfmGI1jMqYJthat there are these kinds of plugins:

  • "river plugins" operating at cluster level
  • "generic plugins" working at node level
  • "index plugins" working at indices level, having access to mappings and
    indices within a cluster
  • "shard plugins" working at shard level, having access to Lucene index
    structures
  • "transport client plugins" working out-of-scope of a cluster

So what should be the kind of plugin I should use? I need to be able to:

  • Run periodically. I plan to use a java Timer for that.
  • Access information about the cluster state and cluster health.

Which other plugin can I look at, to get an example of how to do the
second?

Thanks!!

Flavia

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ah, ok, I guess I am using the generic one. I am overriding the services()
method.
Thanks for your answer!
Flavia

On Friday, March 8, 2013 1:21:08 PM UTC+1, Jörg Prante wrote:

You already have found out about plugin types. Because ES is so
flexible, you can write plugins in many different ways, but there are
some method usages plugins share.

In the class org.elasticsearch.plugins.AbstractPlugin you can see
methods a plugin can activate. The simplest type is the "generic"
plugin, I call it "generic" just because it does not care about indices,
shards, or transport operations, so, it does not use
indexModules/indexServices or shardModules/shardServices methods.

For your task of reading the cluster state and post it somewhere, a
generic plugin is ok.

Jörg

Am 08.03.13 12:47, schrieb fla...@9apps.net <javascript:>:

Still it's not clear to me what kind of plugins you can write. It
seems every plugin I look at is implemented in a different way.
Perhaps there is some documentation about it that I didn't find.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

thanks! I will take a look

On Friday, March 8, 2013 1:26:22 PM UTC+1, Alexander Reelsen wrote:

Hey there,

take a look at https://github.com/spinscale/elasticsearch-graphite-plugin

This plugin already does something similar you seem to try. It is polling
some cluster information every n seconds and pushes them somewhere. It
seems you are trying to do the same, so you might copy some code off it.

--Alex

On Fri, Mar 8, 2013 at 12:47 PM, <fla...@9apps.net <javascript:>> wrote:

Hi!

I made progress with the plugin, it's nearly done. I wrote it using a
service (inheriting AbstractLifecycleComponent), like this one:
https://github.com/elasticsearch/elasticsearch-transport-thrift. The
authors of this plugin also wrote a guide/intro to writing a plugin, see
here: http://jfarrell.github.com/, very nice.

Still it's not clear to me what kind of plugins you can write. It seems
every plugin I look at is implemented in a different way. Perhaps there is
some documentation about it that I didn't find.

Thanks!
Flavia

On Thursday, March 7, 2013 11:24:32 AM UTC+1, fla...@9apps.net wrote:

Hi all,

I'm writing an elasticsearch plugin which periodically (e.g. every
minute) checks the health and state information of a cluster, and posts
this information in Amazon Web Service's CloudWatch. In this way we can
monitor what is going on with our database, and we are able to act as soon
as some of these values are out of the normal range.

I'm pretty much a newbie in elasticsearch, and I read somewhere in another
posthttps://groups.google.com/forum/?hl=en&fromgroups=#!searchin/elasticsearch/writing$20plugin/elasticsearch/fsTVZg3Hyr8/IQfmGI1jMqYJthat there are these kinds of plugins:

  • "river plugins" operating at cluster level
  • "generic plugins" working at node level
  • "index plugins" working at indices level, having access to mappings
    and indices within a cluster
  • "shard plugins" working at shard level, having access to Lucene index
    structures
  • "transport client plugins" working out-of-scope of a cluster

So what should be the kind of plugin I should use? I need to be able to:

  • Run periodically. I plan to use a java Timer for that.
  • Access information about the cluster state and cluster health.

Which other plugin can I look at, to get an example of how to do the
second?

Thanks!!

Flavia

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.