How to implement new 'river' for ES?

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss river:

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?

Thank you, are there sample usages of java bulk api? I'm new to ES so
samples are the best source of info to me :slight_smile:

On 9 янв, 11:45, Karussell tableyourt...@googlemail.com wrote:

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss river:GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?

Hi Nick,

If you read the source code, you will find some tips.

BTW, documentation exists here
: Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH,
David.

Le 9 janvier 2012 à 14:41, Nick Evgeniev nevgeniev@gmail.com a écrit :

Thank you, are there sample usages of java bulk api? I'm new to ES so
samples are the best source of info to me :slight_smile:

On 9 янв, 11:45, Karussell tableyourt...@googlemail.com wrote:

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss
river:GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records
to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?
--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

You don't have to use a river to push data to elasticsearch, you can just
use the APIs to index data to it. I suggest not starting with a River,
the code that pulls data and index it will be the same regardless of where
it runs, so start with it.

On Mon, Jan 9, 2012 at 5:19 PM, david@pilato.fr david@pilato.fr wrote:

**

Hi Nick,

If you read the source code, you will find some tips.

https://github.com/dadoonet/rssriver/blob/master/src/main/java/org/elasticsearch/river/rss/RssRiver.java#L229

BTW, documentation exists here :
Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH,

David.

Le 9 janvier 2012 à 14:41, Nick Evgeniev nevgeniev@gmail.com a écrit :

Thank you, are there sample usages of java bulk api? I'm new to ES so
samples are the best source of info to me :slight_smile:

On 9 янв, 11:45, Karussell tableyourt...@googlemail.com wrote:

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss river:
GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (
www.fixprotocol.org).
Based on what I've read so far I need to implement river to push
records to
ES... I just doubt that json API is performant enough to push
hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to
ES?

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet