When do you index your data?

Elastic Stack Elasticsearch
1

Hi,

I'm new to search and ES. Can anyone discuss when they index their
data? Which method is effective?

In my mind, there's two options:

  1. Parellal with CRUD operations to my relational database

  2. A batch process that runs a SQL statement and then indexes data in
    bulk.

Look forward to hearing from the ES community.

Take care,

Royce

2

It depends on your requirements. If you need real time then method 1,
if not or you want it simple then method 2.

BTW: this project can help https://github.com/Aconex/scrutineer

Peter.

On 9 Dez., 20:16, Royce royce.hay...@gmail.com wrote:

Hi,

I'm new to search and ES. Can anyone discuss when they index their
data? Which method is effective?

In my mind, there's two options:

  1. Parellal with CRUD operations to my relational database

  2. A batch process that runs a SQL statement and then indexes data in
    bulk.

Look forward to hearing from the ES community.

Take care,

Royce

3

For my project, I choose to :
Persist my top entity with Hibernate
Send this top object to an ActiveMQ queue in the same transaction

Then, my jms queue handle some work (pdf rendering, JSON convert and push to ES)

So it's not real time but in most case, near to real time.

BTW, pushing documents to ES is not really the most time consuming (not sure it's the right english expression !). I only make it asynchronous because I have other works to do with my entities.

HTH and hope it makes sense.

David :wink:
@dadoonet

Le 9 déc. 2011 à 20:39, Karussell tableyourtime@googlemail.com a écrit :

It depends on your requirements. If you need real time then method 1,
if not or you want it simple then method 2.

BTW: this project can help https://github.com/Aconex/scrutineer

Peter.

On 9 Dez., 20:16, Royce royce.hay...@gmail.com wrote:

Hi,

I'm new to search and ES. Can anyone discuss when they index their
data? Which method is effective?

In my mind, there's two options:

  1. Parellal with CRUD operations to my relational database

  2. A batch process that runs a SQL statement and then indexes data in
    bulk.

Look forward to hearing from the ES community.

Take care,

Royce