Updating the indexes in every half an hour

Elastic Stack Elasticsearch
1

Dear All,

I am using elasticsearch in some of the my API.
I have created the index and document and have added data in elasticsearch
server from Mysql database.

I am following 3 steps that is,

  1. Delete the index using
    curl -X DELETE 'http://localhost:9200/adminvenue/?pretty=true'

  2. Create the index using
    curl -X PUT 'http://localhost:9200/adminvenue/?pretty=true' -d '
    {.......

  3. Create the mapping using
    curl -X PUT
    'http://localhost:9200/adminvenue/jos_content/_mapping?pretty=true' -d '
    {......

I have set the cron which runs every half an hour.
Cron consist of 2 files named .sh file which is for
creating/deleting/mapping indexes and .php which is use for adding data
from mysql to elasticsearch.

My mysql data is continuously updating that is some data get removed some
modified and some added because of this i have run cron for 1/2 hour.

My concerns is that i have to first delete the indexes and then create the
indexes in every 1/2 hour.
Is there any way to update the indexes that is i just wanted to update the
data in elasticsearch which is added, deleted and modified in mysql ?

Any suggestion is very much appreciated.

Thanks,
Vallabh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/718b0eb6-048e-46bf-95f4-683389d24496%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You can use JDBC river plugin to fetch from database directly.

On Monday, February 10, 2014 8:40:31 PM UTC+11, Vallabh Bothre wrote:

Dear All,

I am using elasticsearch in some of the my API.
I have created the index and document and have added data in elasticsearch
server from Mysql database.

I am following 3 steps that is,

  1. Delete the index using
    curl -X DELETE 'http://localhost:9200/adminvenue/?pretty=true'

  2. Create the index using
    curl -X PUT 'http://localhost:9200/adminvenue/?pretty=true' -d '
    {.......

  3. Create the mapping using
    curl -X PUT '
    http://localhost:9200/adminvenue/jos_content/_mapping?pretty=true' -d '
    {......

I have set the cron which runs every half an hour.
Cron consist of 2 files named .sh file which is for
creating/deleting/mapping indexes and .php which is use for adding data
from mysql to elasticsearch.

My mysql data is continuously updating that is some data get removed some
modified and some added because of this i have run cron for 1/2 hour.

My concerns is that i have to first delete the indexes and then create the
indexes in every 1/2 hour.
Is there any way to update the indexes that is i just wanted to update the
data in elasticsearch which is added, deleted and modified in mysql ?

Any suggestion is very much appreciated.

Thanks,
Vallabh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/05415e13-f76c-420e-bac8-15a754230296%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thank you Kevin for your reply,

But i was searching for the command which will directly update the data
without deleting and creating the indexes.

Is there any command like curl -X UPDATE in elasticsearch ?
which will not add already existing data in elastic.

On Monday, February 10, 2014 3:38:28 PM UTC+5:30, Kevin Wang wrote:

You can use JDBC river plugin to fetch from database directly.

GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch

On Monday, February 10, 2014 8:40:31 PM UTC+11, Vallabh Bothre wrote:

Dear All,

I am using elasticsearch in some of the my API.
I have created the index and document and have added data in
elasticsearch server from Mysql database.

I am following 3 steps that is,

  1. Delete the index using
    curl -X DELETE 'http://localhost:9200/adminvenue/?pretty=true'

  2. Create the index using
    curl -X PUT 'http://localhost:9200/adminvenue/?pretty=true' -d '
    {.......

  3. Create the mapping using
    curl -X PUT '
    http://localhost:9200/adminvenue/jos_content/_mapping?pretty=true' -d '
    {......

I have set the cron which runs every half an hour.
Cron consist of 2 files named .sh file which is for
creating/deleting/mapping indexes and .php which is use for adding data
from mysql to elasticsearch.

My mysql data is continuously updating that is some data get removed some
modified and some added because of this i have run cron for 1/2 hour.

My concerns is that i have to first delete the indexes and then create
the indexes in every 1/2 hour.
Is there any way to update the indexes that is i just wanted to update
the data in elasticsearch which is added, deleted and modified in mysql ?

Any suggestion is very much appreciated.

Thanks,
Vallabh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1f8c781f-2a67-48ee-8f77-bd38a3d8a40e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Vallabh,

You can do full document updates/replace by simply using POST
http://server:9200///id. Just specify the document ID of the
document you want to update and it will replace that document.

You can also do partial updates of an existing document like this:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db5dd077-5af5-4b0b-aef6-6f080141a6f7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

You can use an alias to enable switching between two indexes. Therefore,
you can build a new index while allowing searching against the old one.
When the new one is successfully built change the alias to point to the new
one.

On Mon, Feb 10, 2014 at 1:40 AM, Vallabh Bothre vallabh.bothre@gmail.comwrote:

Dear All,

I am using elasticsearch in some of the my API.
I have created the index and document and have added data in elasticsearch
server from Mysql database.

I am following 3 steps that is,

  1. Delete the index using
    curl -X DELETE 'http://localhost:9200/adminvenue/?pretty=true'

  2. Create the index using
    curl -X PUT 'http://localhost:9200/adminvenue/?pretty=true' -d '
    {.......

  3. Create the mapping using
    curl -X PUT '
    http://localhost:9200/adminvenue/jos_content/_mapping?pretty=true' -d '
    {......

I have set the cron which runs every half an hour.
Cron consist of 2 files named .sh file which is for
creating/deleting/mapping indexes and .php which is use for adding data
from mysql to elasticsearch.

My mysql data is continuously updating that is some data get removed some
modified and some added because of this i have run cron for 1/2 hour.

My concerns is that i have to first delete the indexes and then create the
indexes in every 1/2 hour.
Is there any way to update the indexes that is i just wanted to update the
data in elasticsearch which is added, deleted and modified in mysql ?

Any suggestion is very much appreciated.

Thanks,
Vallabh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/718b0eb6-048e-46bf-95f4-683389d24496%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFjHw35KEsrd6J_pC3SjtWx5KbV5xKPBXpvPNzsNGff6d45S7w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Thanks RKM.

On Tuesday, February 11, 2014 12:36:59 AM UTC+5:30, RKM wrote:

You can use an alias to enable switching between two indexes. Therefore,
you can build a new index while allowing searching against the old one.
When the new one is successfully built change the alias to point to the new
one.

On Mon, Feb 10, 2014 at 1:40 AM, Vallabh Bothre <vallabh...@gmail.com<javascript:>

wrote:

Dear All,

I am using elasticsearch in some of the my API.
I have created the index and document and have added data in
elasticsearch server from Mysql database.

I am following 3 steps that is,

  1. Delete the index using
    curl -X DELETE 'http://localhost:9200/adminvenue/?pretty=true'

  2. Create the index using
    curl -X PUT 'http://localhost:9200/adminvenue/?pretty=true' -d '
    {.......

  3. Create the mapping using
    curl -X PUT '
    http://localhost:9200/adminvenue/jos_content/_mapping?pretty=true' -d '
    {......

I have set the cron which runs every half an hour.
Cron consist of 2 files named .sh file which is for
creating/deleting/mapping indexes and .php which is use for adding data
from mysql to elasticsearch.

My mysql data is continuously updating that is some data get removed some
modified and some added because of this i have run cron for 1/2 hour.

My concerns is that i have to first delete the indexes and then create
the indexes in every 1/2 hour.
Is there any way to update the indexes that is i just wanted to update
the data in elasticsearch which is added, deleted and modified in mysql ?

Any suggestion is very much appreciated.

Thanks,
Vallabh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/718b0eb6-048e-46bf-95f4-683389d24496%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/82a37167-7061-42f6-88f3-8bf89ce07c5c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

For streams (changing data over time with timestamps), the simplest method
in ES is really to manage timeframe-based indexes, like in the logstash
model. So, you can decide to drop indexes when they become too old. There
are tools that can help in this job, the curator.

By using index aliases, there is no disadvantage in comparison to a single
index. And, ES index drops are very fast, and they do not require resources
like document deletes which allocate space in segments. This should be
carefully considered.

With JDBC river, there is no requirement to delete and create the indexes,
just use the same _id to address your docs, by a crontab definition. It is
very easy to run this every 30 minutes. The docs will get overwritten and
receive a new version number.

Aside form the river, by checking the version of the docs in ES, you can
find out old documents. The JDBC river does no longer take the job of
deleting old documents, this has to be done from outside.

Jörg

On Mon, Feb 10, 2014 at 2:41 PM, Vallabh Bothre vallabh.bothre@gmail.comwrote:

Thank you Kevin for your reply,

But i was searching for the command which will directly update the data
without deleting and creating the indexes.

Is there any command like curl -X UPDATE in elasticsearch ?
which will not add already existing data in elastic.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGMWY78d_qgt-P8VQCV9inDMKZUgpBir3_APh%3DHERpHw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.