JDBC Update policy issue

Elastic Stack Elasticsearch
#1

Hi All,

I'm trying ES + MySQL with JDBC plugin. I have created a small table in
MySQL

CREATE TABLE users (

id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,

name VARCHAR(100),

department VARCHAR(100),

created TIMESTAMP DEFAULT NOW()

);

I made sample river with CURL

curl -XPUT 'localhost:9200/samdoc/' -d '{}'

curl -XPUT 'http://locakhost:9200/_river/dbusers/_meta' -d '{

"type" : "jdbc",

"jdbc" : {

"strategy" : "simple",

"poll" : "10m",

    "driver" : "com.mysql.jdbc.Driver",

    "url" : "jdbc:mysql://localhost:3306/test",

    "user" : "",

    "password" : "",

    "sql" : "select * , id as _id from users"

},

"index" : {

    "index" : "dbusers",

    "type" : "jdbc",

"versioning" : true

}

}'

ES index the SQL table and polls the data at every 2 min. Each polling
request is making the version update in index. I have updated the database
table and on next poll the ES index didn't reflect the changes. I also
tried to refresh the index, but no updates were available in the ES.

I have restarted the ES cluster for checking & noticed two things:
1: The ES index get created with new value and version is 1.
2: If any changes are made in the SQL during ES downtime time, entire table
is index again and all previous version are deleted.

How to reflect the updates in the table to the ES index? Did i miss
something?

Thanks
Ajit

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

#2

Thanks for repoting. There is an issue in the update mechanism, I have
to look into it. But it is correct that all values are indexed again and
old documents get deleted, there is no other way to provide the "newest"
data to ES. Therefore I will better document it and add a river strategy
which can handle incremental updates (with more SQL statements).

For bettertracing, I opened an issue for this

Jörg

Am 29.03.13 13:55, schrieb Ajitpal Singh:

Hi All,

I'm trying ES + MySQL with JDBC plugin. I have created a small table
in MySQL

    CREATE TABLE users (

    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,

    name VARCHAR(100),

    department VARCHAR(100),

    created TIMESTAMP DEFAULT NOW()

    );

I made sample river with CURL

  curl -XPUT 'localhost:9200/samdoc/' -d '{}'

    curl -XPUT 'http://locakhost:9200/_river/dbusers/_meta' -d '{ 

        "type" : "jdbc",

        "jdbc" : {

    "strategy" : "simple",

    "poll" : "10m",

            "driver" : "com.mysql.jdbc.Driver",

            "url" : "jdbc:mysql://localhost:3306/test",

            "user" : "",

            "password" : "",

            "sql" : "select * , id as _id from users"

        },

        "index" : {

            "index" : "dbusers",

            "type" : "jdbc",

    "versioning" : true

        }

    }'

ES index the SQL table and polls the data at every 2 min. Each polling
request is making the version update in index. I have updated the
database table and on next poll the ES index didn't reflect the
changes. I also tried to refresh the index, but no updates were
available in the ES.

I have restarted the ES cluster for checking & noticed two things:
1: The ES index get created with new value and version is 1.
2: If any changes are made in the SQL during ES downtime time, entire
table is index again and all previous version are deleted.

How to reflect the updates in the table to the ES index? Did i miss
something?

Thanks
Ajit

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

#3