How to implement new 'river' for ES?

Elastic Stack Elasticsearch
1

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?

2

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss river:

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?

3

Thank you, are there sample usages of java bulk api? I'm new to ES so
samples are the best source of info to me :slight_smile:

On 9 янв, 11:45, Karussell tableyourt...@googlemail.com wrote:

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss river:GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?

4

Hi Nick,

If you read the source code, you will find some tips.

BTW, documentation exists here
: Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH,
David.

Le 9 janvier 2012 à 14:41, Nick Evgeniev nevgeniev@gmail.com a écrit :

Thank you, are there sample usages of java bulk api? I'm new to ES so
samples are the best source of info to me :slight_smile:

On 9 янв, 11:45, Karussell tableyourt...@googlemail.com wrote:

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss
river:GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (www.fixprotocol.org).
Based on what I've read so far I need to implement river to push records
to
ES... I just doubt that json API is performant enough to push hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to ES?
--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

5

You don't have to use a river to push data to elasticsearch, you can just
use the APIs to index data to it. I suggest not starting with a River,
the code that pulls data and index it will be the same regardless of where
it runs, so start with it.

On Mon, Jan 9, 2012 at 5:19 PM, david@pilato.fr david@pilato.fr wrote:

**

Hi Nick,

If you read the source code, you will find some tips.

https://github.com/dadoonet/rssriver/blob/master/src/main/java/org/elasticsearch/river/rss/RssRiver.java#L229

BTW, documentation exists here :
Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH,

David.

Le 9 janvier 2012 à 14:41, Nick Evgeniev nevgeniev@gmail.com a écrit :

Thank you, are there sample usages of java bulk api? I'm new to ES so
samples are the best source of info to me :slight_smile:

On 9 янв, 11:45, Karussell tableyourt...@googlemail.com wrote:

You can use json (bulk api) or Java's bulk api.

For the river you could have a look into the rss river:
GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)

Peter.

On 8 Jan., 23:56, Nick Evgeniev nevgen...@gmail.com wrote:

Hi,

I'm thinking about using ES to index FIX messages (
www.fixprotocol.org).
Based on what I've read so far I need to implement river to push
records to
ES... I just doubt that json API is performant enough to push
hundreds of
thousands of FIX messages per second to ES... Is there
howto/template/working example of how to do this?

Also, is implementing a river is the fastest way of pushing data to
ES?

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet