How can I input data from java to ES in real time?

Marian_Valero · January 7, 2015, 1:46pm

I'm reading data from my oracle database to java with JDBC but I want to
know how can I input the data that I'm getting in Elasticsearch in real
time.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8010611-a842-4ab0-aad9-62f1edf747a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 7, 2015, 1:53pm

I'd recommend looking at GitHub - dadoonet/legacy-search: Demo project showing how to add elasticsearch to a legacy application.

Branch 01, basically does this: https://github.com/dadoonet/legacy-search/blob/01-direct/src/main/java/fr/pilato/demo/legacysearch/dao/ElasticsearchDao.java#L44

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 janv. 2015 à 14:46, Marian Valero marianv0992@gmail.com a écrit :

I'm reading data from my oracle database to java with JDBC but I want to know how can I input the data that I'm getting in Elasticsearch in real time.

Thanks.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8010611-a842-4ab0-aad9-62f1edf747a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/22ECDD92-5671-4EFE-A3DF-681C6D8D2FCC%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 7, 2015, 3:41pm

I have this:

public static void main(String args[]) throws SQLException {

int count=0;

    String url = "jdbc:oracle:thin:eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    Timestamp timestamp = null;
    
    while(result.next() && count<1000){
    count++;
    String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");
          
        System.out.println("Responselog from Oracle : "+

result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}
System.out.println("done");
}

I'm reading 1000 registers of the table and I want to insert this in
Elasticsearch, but this is an example, I want to do that for all registers
of my database, this is millions of registers. I don't understand very well
that you send me.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8391d264-86c0-48e5-ac90-613cc9f59628%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 7, 2015, 4:08pm

You should read this: Elasticsearch Platform — Find real-time answers at scale | Elastic http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/index_.html

You need to create a Client: Elasticsearch Platform — Find real-time answers at scale | Elastic
http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.htmlYou need to create a JSON from your properties. See: Elasticsearch Platform — Find real-time answers at scale | Elastic http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/index_.html#helpers
Then you should use a Bulk to index all those documents: See: Elasticsearch Platform — Find real-time answers at scale | Elastic http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/bulk.html
Something like this:
BulkRequestBuilder bulkRequest = client.prepareBulk();
while(result.next() && count<1000){
// YOUR ORACLE stuff here
// either use client#prepare, or use Requests# to directly build index/delete requests
bulkRequest.add(client.prepareIndex("indexname", "typename", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid) // Add your other fields
.endObject()
)
);
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
Do that every 1000 or 10000 documents. Don’t send too many requests within the same bulk. It will use too much memory and probably will slow down your indexation rate.

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 7 janv. 2015 à 16:41, Marian Valero marianv0992@gmail.com a écrit :

I have this:

public static void main(String args) throws SQLException {
	int count=0;
	
    String url = "jdbc:oracle:thin:eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    Timestamp timestamp = null;
    
    while(result.next() && count<1000){
    	count++;
    	String id = result.getString("id");
    	String deliveryid = result.getString("deliveryid");
    	String msgid = result.getString("msgid");
    	String rspdate = result.getString("rspdate");
    	String parsedresponse = result.getString("parsedresponse");
    	String shortcode = result.getString("shortcode");
    	String insid = result.getString("insid");
    	String mobilenumber = result.getString("mobilenumber");
    	String rawresponse = result.getString("rawresponse");
         	
        System.out.println("Responselog from Oracle : "+ result.getString("id")+" "+result.getString("deliveryid")+" "+result.getString("msgid")+" "+result.getString("rspdate")+" "+result.getString("parsedresponse")+" "+result.getString("shortcode")+" "+result.getString("insid")+" "+result.getString("mobilenumber")+" "+result.getString("rawresponse")); 
    }               
    System.out.println("done");
}
I'm reading 1000 registers of the table and I want to insert this in Elasticsearch, but this is an example, I want to do that for all registers of my database, this is millions of registers. I don't understand very well that you send me.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8391d264-86c0-48e5-ac90-613cc9f59628%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/8391d264-86c0-48e5-ac90-613cc9f59628%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A69A19D5-CEC2-4A3E-9CBC-5024438288AE%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 7, 2015, 4:28pm

Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I do
it that way:

public static void main(String args[]) throws SQLException {
     
int count=0;
Node node = nodeBuilder().client(true).node();
Client client = node.client();

    String url = "jdbc:oracle:thin:@eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    // Timestamp timestamp = null;
    
  BulkRequestBuilder bulkRequest = client.prepareBulk();
    
    while(result.next() && count<1000){
    count++;
        String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");
          
    XContentBuilder builder = jsonBuilder()
    .startObject()
            .field("deliveryid", deliveryid)
            .field("msgid", msgid)
            .field("message", "trying out Elasticsearch")
        .endObject();


    // YOUR ORACLE stuff here
    // either use client#prepare, or use Requests# to directly build

index/delete requests
bulkRequest.add(client.prepareIndex("logs", "responselog", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid)
.field("rspdate", rspdate)
.field("parsedresponse", parsedresponse)
.field("shortcode", shortcode)
.field("insid", insid)
.field("mobilenumber", mobilenumber)
.field("rawresponse", rawresponse)
.endObject()
)
);

        System.out.println("Responselog from Oracle : "+

result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}

BulkResponse bulkResponse = bulkRequest.execute().actionGet();
System.out.println("done");
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · January 7, 2015, 4:35pm

You can use the JDBC plugin instead of reinventing the wheel.

Jörg

On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero marianv0992@gmail.com wrote:

Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I do
it that way:
public static void main(String args[]) throws SQLException {

int count=0;
Node node = nodeBuilder().client(true).node();
Client client = node.client();

    String url = "jdbc:oracle:thin:@eso:1521:eso";

    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");

    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();

    // Timestamp timestamp = null;

  BulkRequestBuilder bulkRequest = client.prepareBulk();

    while(result.next() && count<1000){
    count++;
        String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");

    XContentBuilder builder = jsonBuilder()
    .startObject()
            .field("deliveryid", deliveryid)
            .field("msgid", msgid)
            .field("message", "trying out Elasticsearch")
        .endObject();


    // YOUR ORACLE stuff here
    // either use client#prepare, or use Requests# to directly build
index/delete requests
bulkRequest.add(client.prepareIndex("logs", "responselog", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid)
.field("rspdate", rspdate)
.field("parsedresponse", parsedresponse)
.field("shortcode", shortcode)
.field("insid", insid)
.field("mobilenumber", mobilenumber)
.field("rawresponse", rawresponse)
.endObject()
)
);
        System.out.println("Responselog from Oracle : "+
result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
System.out.println("done");
}
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 7, 2015, 4:39pm

Hi Jörg I have problems when I use that, I didn't insert the data, I create
the river but I can get the data in ES.

El miércoles, 7 de enero de 2015 12:05:47 UTC-4:30, Jörg Prante escribió:

You can use the JDBC plugin instead of reinventing the wheel.

GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch

Jörg

On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero <maria...@gmail.com
<javascript:>> wrote:
Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I
do it that way:
public static void main(String args[]) throws SQLException {
     
int count=0;
Node node = nodeBuilder().client(true).node();
Client client = node.client();

    String url = "jdbc:oracle:thin:@eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    // Timestamp timestamp = null;
    
  BulkRequestBuilder bulkRequest = client.prepareBulk();
    
    while(result.next() && count<1000){
    count++;
        String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");
          
    XContentBuilder builder = jsonBuilder()
    .startObject()
            .field("deliveryid", deliveryid)
            .field("msgid", msgid)
            .field("message", "trying out Elasticsearch")
        .endObject();


    // YOUR ORACLE stuff here
    // either use client#prepare, or use Requests# to directly build 
index/delete requests
bulkRequest.add(client.prepareIndex("logs", "responselog", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid)
.field("rspdate", rspdate)
.field("parsedresponse", parsedresponse)
.field("shortcode", shortcode)
.field("insid", insid)
.field("mobilenumber", mobilenumber)
.field("rawresponse", rawresponse)
.endObject()
)
);
        System.out.println("Responselog from Oracle : "+ 
result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
System.out.println("done");
} 
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d90b58c2-fb33-4dc3-b630-183c12e0566c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · January 7, 2015, 4:48pm

Are there any error messages? What did you try to insert data?

Jörg

On Wed, Jan 7, 2015 at 5:39 PM, Marian Valero marianv0992@gmail.com wrote:

Hi Jörg I have problems when I use that, I didn't insert the data, I
create the river but I can get the data in ES.

El miércoles, 7 de enero de 2015 12:05:47 UTC-4:30, Jörg Prante escribió:
You can use the JDBC plugin instead of reinventing the wheel.

GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch

Jörg

On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero maria...@gmail.com wrote:
Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I
do it that way:
public static void main(String args[]) throws SQLException {

int count=0;
Node node = nodeBuilder().client(true).node();
Client client = node.client();

    String url = "jdbc:oracle:thin:@eso:1521:eso";

    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");

    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();

    // Timestamp timestamp = null;

  BulkRequestBuilder bulkRequest = client.prepareBulk();

    while(result.next() && count<1000){
    count++;
        String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");

    XContentBuilder builder = jsonBuilder()
    .startObject()
            .field("deliveryid", deliveryid)
            .field("msgid", msgid)
            .field("message", "trying out Elasticsearch")
        .endObject();


    // YOUR ORACLE stuff here
    // either use client#prepare, or use Requests# to directly
build index/delete requests
bulkRequest.add(client.prepareIndex("logs", "responselog", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid)
.field("rspdate", rspdate)
.field("parsedresponse", parsedresponse)
.field("shortcode", shortcode)
.field("insid", insid)
.field("mobilenumber", mobilenumber)
.field("rawresponse", rawresponse)
.endObject()
)
);
        System.out.println("Responselog from Oracle : "+
result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
System.out.println("done");
}
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d90b58c2-fb33-4dc3-b630-183c12e0566c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d90b58c2-fb33-4dc3-b630-183c12e0566c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFkE%2BO_b6TstTgOZN8VMyrTxJg_dCxic3XpfyYt8V0c6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 7, 2015, 4:50pm

Of course, but if you read carefully the links to the doc I pasted, you will see that you have to import a class.

Doc says:
import static org.elasticsearch.common.xcontent.XContentFactory.*;

XContentBuilder builder = jsonBuilder()
.startObject()
.field("user", "kimchy")
.field("postDate", new Date())
.field("message", "trying out Elasticsearch")
.endObject()

Did you see it?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 7 janv. 2015 à 17:28, Marian Valero marianv0992@gmail.com a écrit :

Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I do it that way:

public static void main(String args[]) throws SQLException {
	    	
	int count=0;
	Node node = nodeBuilder().client(true).node();
	Client client = node.client();
	
    String url = "jdbc:oracle:thin:@eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    // Timestamp timestamp = null;
    
 	BulkRequestBuilder bulkRequest = client.prepareBulk();
    
    while(result.next() && count<1000){
    	count++;
        String id = result.getString("id");
    	String deliveryid = result.getString("deliveryid");
    	String msgid = result.getString("msgid");
    	String rspdate = result.getString("rspdate");
    	String parsedresponse = result.getString("parsedresponse");
    	String shortcode = result.getString("shortcode");
    	String insid = result.getString("insid");
    	String mobilenumber = result.getString("mobilenumber");
    	String rawresponse = result.getString("rawresponse");
          	
    	XContentBuilder builder = jsonBuilder()
    			.startObject()
    		        .field("deliveryid", deliveryid)
    		        .field("msgid", msgid)
    		        .field("message", "trying out Elasticsearch")
    		    .endObject();


    	// YOUR ORACLE stuff here
    	// either use client#prepare, or use Requests# to directly build index/delete requests
    	bulkRequest.add(client.prepareIndex("logs", "responselog", id)
    	        .setSource(jsonBuilder()
    	                    .startObject()
    	                        .field("deliveryid", deliveryid)
    	                        .field("msgid", msgid) 
    	                        .field("rspdate", rspdate)
    	                        .field("parsedresponse", parsedresponse)
    	                        .field("shortcode", shortcode)
    	                        .field("insid", insid)
    	                        .field("mobilenumber", mobilenumber)
    	                        .field("rawresponse", rawresponse)
    	                    .endObject()
    	                  )
    	        );    	
         	
        System.out.println("Responselog from Oracle : "+ result.getString("id")+" "+result.getString("deliveryid")+" "+result.getString("msgid")+" "+result.getString("rspdate")+" "+result.getString("parsedresponse")+" "+result.getString("shortcode")+" "+result.getString("insid")+" "+result.getString("mobilenumber")+" "+result.getString("rawresponse")); 
    }              
    
	BulkResponse bulkResponse = bulkRequest.execute().actionGet();
	System.out.println("done");
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/568DFD50-81A0-46EF-9B64-1EEE5159BDE5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 7, 2015, 4:52pm

Well. I did not mention your plugin because he basically asked for "I want to know how can I input the data that I'm getting in Elasticsearch in real time.".
Real time aspect here seems to be important.

Elasticsearch is not real time but near real time.
JDBC river is obviously not real time as IIRC it runs a job every x minutes or so.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 7 janv. 2015 à 17:35, joergprante@gmail.com a écrit :

You can use the JDBC plugin instead of reinventing the wheel.

GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch https://github.com/jprante/elasticsearch-river-jdbc

Jörg

On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero <marianv0992@gmail.com mailto:marianv0992@gmail.com> wrote:
Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I do it that way:
public static void main(String args[]) throws SQLException {
	    	
	int count=0;
	Node node = nodeBuilder().client(true).node();
	Client client = node.client();
	
    String url = "jdbc:oracle:thin:@eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    // Timestamp timestamp = null;
    
 	BulkRequestBuilder bulkRequest = client.prepareBulk();
    
    while(result.next() && count<1000){
    	count++;
        String id = result.getString("id");
    	String deliveryid = result.getString("deliveryid");
    	String msgid = result.getString("msgid");
    	String rspdate = result.getString("rspdate");
    	String parsedresponse = result.getString("parsedresponse");
    	String shortcode = result.getString("shortcode");
    	String insid = result.getString("insid");
    	String mobilenumber = result.getString("mobilenumber");
    	String rawresponse = result.getString("rawresponse");
          	
    	XContentBuilder builder = jsonBuilder()
    			.startObject()
    		        .field("deliveryid", deliveryid)
    		        .field("msgid", msgid)
    		        .field("message", "trying out Elasticsearch")
    		    .endObject();


    	// YOUR ORACLE stuff here
    	// either use client#prepare, or use Requests# to directly build index/delete requests
    	bulkRequest.add(client.prepareIndex("logs", "responselog", id)
    	        .setSource(jsonBuilder()
    	                    .startObject()
    	                        .field("deliveryid", deliveryid)
    	                        .field("msgid", msgid) 
    	                        .field("rspdate", rspdate)
    	                        .field("parsedresponse", parsedresponse)
    	                        .field("shortcode", shortcode)
    	                        .field("insid", insid)
    	                        .field("mobilenumber", mobilenumber)
    	                        .field("rawresponse", rawresponse)
    	                    .endObject()
    	                  )
    	        );    	
         	
        System.out.println("Responselog from Oracle : "+ result.getString("id")+" "+result.getString("deliveryid")+" "+result.getString("msgid")+" "+result.getString("rspdate")+" "+result.getString("parsedresponse")+" "+result.getString("shortcode")+" "+result.getString("insid")+" "+result.getString("mobilenumber")+" "+result.getString("rawresponse")); 
    }              
    
	BulkResponse bulkResponse = bulkRequest.execute().actionGet();
	System.out.println("done");
} 
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 7, 2015, 5:24pm

Yes! I previously import this but the problem persist

El miércoles, 7 de enero de 2015 12:22:39 UTC-4:30, David Pilato escribió:

Of course, but if you read carefully the links to the doc I pasted, you
will see that you have to import a class.

Doc says:

import static org.elasticsearch.common.xcontent.XContentFactory.*;

XContentBuilder builder = jsonBuilder()
.startObject()
.field("user", "kimchy")
.field("postDate", new Date())
.field("message", "trying out Elasticsearch")
.endObject()

Did you see it?

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 7 janv. 2015 à 17:28, Marian Valero <maria...@gmail.com <javascript:>>
a écrit :

Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I do
it that way:
public static void main(String args[]) throws SQLException {
     
int count=0;
Node node = nodeBuilder().client(true).node();
Client client = node.client();

    String url = "jdbc:oracle:thin:@eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    // Timestamp timestamp = null;
    
  BulkRequestBuilder bulkRequest = client.prepareBulk();
    
    while(result.next() && count<1000){
    count++;
        String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");
          
    XContentBuilder builder = jsonBuilder()
    .startObject()
            .field("deliveryid", deliveryid)
            .field("msgid", msgid)
            .field("message", "trying out Elasticsearch")
        .endObject();


    // YOUR ORACLE stuff here
    // either use client#prepare, or use Requests# to directly build 
index/delete requests
bulkRequest.add(client.prepareIndex("logs", "responselog", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid)
.field("rspdate", rspdate)
.field("parsedresponse", parsedresponse)
.field("shortcode", shortcode)
.field("insid", insid)
.field("mobilenumber", mobilenumber)
.field("rawresponse", rawresponse)
.endObject()
)
);
        System.out.println("Responselog from Oracle : "+ 
result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
System.out.println("done");
} 
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/73ca9160-f1b4-4ccc-808b-d640082f6ddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 7, 2015, 5:31pm

I think that is problem of the jar but I import this well.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63744158-4ecc-44cb-862c-29cec2097b11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · January 7, 2015, 5:47pm

The definitions of "real time" are quite vague. I also tend to use "real
time" for sub-second latencies with a guarantee of a maximum response time.
Other just mean by "real time" everything that is not batching, pushing
data over the wire as the changes happen, no matter how large the change is

a process which can take huge delays in worst case, even minutes.

Although it is possible to run a select query every second with JDBC plugin
crontab, this puts a heavy load on the system with not much gain, so I
don't recommend it.

For instant replication, one would have to extend JDBC plugin with a
special queue on the side of the RDBMS and use proprietary driver API.
Even with such a queue the latency is often higher than a few seconds.

Jörg

On Wed, Jan 7, 2015 at 5:52 PM, David Pilato david@pilato.fr wrote:

Well. I did not mention your plugin because he basically asked for "I want
to know how can I input the data that I'm getting in Elasticsearch in real
time.".
Real time aspect here seems to be important.

Elasticsearch is not real time but near real time.
JDBC river is obviously not real time as IIRC it runs a job every x
minutes or so.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 7 janv. 2015 à 17:35, joergprante@gmail.com a écrit :

You can use the JDBC plugin instead of reinventing the wheel.

GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch

Jörg

On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero marianv0992@gmail.com
wrote:
Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I
do it that way:
public static void main(String args[]) throws SQLException {

int count=0;
Node node = nodeBuilder().client(true).node();
Client client = node.client();

    String url = "jdbc:oracle:thin:@eso:1521:eso";

    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");

    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();

    // Timestamp timestamp = null;

  BulkRequestBuilder bulkRequest = client.prepareBulk();

    while(result.next() && count<1000){
    count++;
        String id = result.getString("id");
    String deliveryid = result.getString("deliveryid");
    String msgid = result.getString("msgid");
    String rspdate = result.getString("rspdate");
    String parsedresponse = result.getString("parsedresponse");
    String shortcode = result.getString("shortcode");
    String insid = result.getString("insid");
    String mobilenumber = result.getString("mobilenumber");
    String rawresponse = result.getString("rawresponse");

    XContentBuilder builder = jsonBuilder()
    .startObject()
            .field("deliveryid", deliveryid)
            .field("msgid", msgid)
            .field("message", "trying out Elasticsearch")
        .endObject();


    // YOUR ORACLE stuff here
    // either use client#prepare, or use Requests# to directly build
index/delete requests
bulkRequest.add(client.prepareIndex("logs", "responselog", id)
.setSource(jsonBuilder()
.startObject()
.field("deliveryid", deliveryid)
.field("msgid", msgid)
.field("rspdate", rspdate)
.field("parsedresponse", parsedresponse)
.field("shortcode", shortcode)
.field("insid", insid)
.field("mobilenumber", mobilenumber)
.field("rawresponse", rawresponse)
.endObject()
)
);
        System.out.println("Responselog from Oracle : "+
result.getString("id")+" "+result.getString("deliveryid")+"
"+result.getString("msgid")+" "+result.getString("rspdate")+"
"+result.getString("parsedresponse")+" "+result.getString("shortcode")+"
"+result.getString("insid")+" "+result.getString("mobilenumber")+"
"+result.getString("rawresponse"));
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
System.out.println("done");
}
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE0CUreVbvpQrtXE186yqjA%2BtiYPCfZkwK_3QxfoOnOSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 7, 2015, 5:56pm

Agreed. Reading is database could be an heavy operation.
That’s why I always recommend if possible to send to elasticsearch data in the same « transaction » as you persist your data to the database.
You often have a bean ready to be persisted in memory. Serializing to JSON with Jackson and then sending to elasticsearch does not cost much in term of CPU usage/memory and in term of line of codes.

But for that, you need to have an access to the application which produces the data.

If not, I agree, you should not reinvent the wheel and use whatever ETL you want to read, transform and index data. Such as JDBC River.
My advice about JDBC river would be to run it outside elasticsearch as a feeder: GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch https://github.com/jprante/elasticsearch-river-jdbc#two-flavors-river-or-feeder

Best

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 7 janv. 2015 à 18:47, joergprante@gmail.com a écrit :

The definitions of "real time" are quite vague. I also tend to use "real time" for sub-second latencies with a guarantee of a maximum response time. Other just mean by "real time" everything that is not batching, pushing data over the wire as the changes happen, no matter how large the change is - a process which can take huge delays in worst case, even minutes.

Although it is possible to run a select query every second with JDBC plugin crontab, this puts a heavy load on the system with not much gain, so I don't recommend it.

For instant replication, one would have to extend JDBC plugin with a special queue on the side of the RDBMS and use proprietary driver API. Even with such a queue the latency is often higher than a few seconds.

Jörg

On Wed, Jan 7, 2015 at 5:52 PM, David Pilato <david@pilato.fr mailto:david@pilato.fr> wrote:
Well. I did not mention your plugin because he basically asked for "I want to know how can I input the data that I'm getting in Elasticsearch in real time.".
Real time aspect here seems to be important.

Elasticsearch is not real time but near real time.
JDBC river is obviously not real time as IIRC it runs a job every x minutes or so.

--
David Pilato | Technical Advocate | Elasticsearch.com http://elasticsearch.com/
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs
Le 7 janv. 2015 à 17:35, joergprante@gmail.com mailto:joergprante@gmail.com a écrit :

You can use the JDBC plugin instead of reinventing the wheel.

GitHub - jprante/elasticsearch-jdbc: JDBC importer for Elasticsearch https://github.com/jprante/elasticsearch-river-jdbc

Jörg

On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero <marianv0992@gmail.com mailto:marianv0992@gmail.com> wrote:
Thanks!

I have a problem with jsonBuilder() method, eclipse don't found this, I do it that way:
public static void main(String args[]) throws SQLException {
	    	
	int count=0;
	Node node = nodeBuilder().client(true).node();
	Client client = node.client();
	
    String url = "jdbc:oracle:thin:@eso:1521:eso"; 
  
    Properties props = new Properties();
    props.setProperty("user", "eso");
    props.setProperty("password", "eso");
  
    Connection conn = DriverManager.getConnection(url,props);

    String sql ="select * from responselog";

    PreparedStatement preStatement = conn.prepareStatement(sql);

    ResultSet result = preStatement.executeQuery();
  
    // Timestamp timestamp = null;
    
 	BulkRequestBuilder bulkRequest = client.prepareBulk();
    
    while(result.next() && count<1000){
    	count++;
        String id = result.getString("id");
    	String deliveryid = result.getString("deliveryid");
    	String msgid = result.getString("msgid");
    	String rspdate = result.getString("rspdate");
    	String parsedresponse = result.getString("parsedresponse");
    	String shortcode = result.getString("shortcode");
    	String insid = result.getString("insid");
    	String mobilenumber = result.getString("mobilenumber");
    	String rawresponse = result.getString("rawresponse");
          	
    	XContentBuilder builder = jsonBuilder()
    			.startObject()
    		        .field("deliveryid", deliveryid)
    		        .field("msgid", msgid)
    		        .field("message", "trying out Elasticsearch")
    		    .endObject();


    	// YOUR ORACLE stuff here
    	// either use client#prepare, or use Requests# to directly build index/delete requests
    	bulkRequest.add(client.prepareIndex("logs", "responselog", id)
    	        .setSource(jsonBuilder()
    	                    .startObject()
    	                        .field("deliveryid", deliveryid)
    	                        .field("msgid", msgid) 
    	                        .field("rspdate", rspdate)
    	                        .field("parsedresponse", parsedresponse)
    	                        .field("shortcode", shortcode)
    	                        .field("insid", insid)
    	                        .field("mobilenumber", mobilenumber)
    	                        .field("rawresponse", rawresponse)
    	                    .endObject()
    	                  )
    	        );    	
         	
        System.out.println("Responselog from Oracle : "+ result.getString("id")+" "+result.getString("deliveryid")+" "+result.getString("msgid")+" "+result.getString("rspdate")+" "+result.getString("parsedresponse")+" "+result.getString("shortcode")+" "+result.getString("insid")+" "+result.getString("mobilenumber")+" "+result.getString("rawresponse")); 
    }              
    
	BulkResponse bulkResponse = bulkRequest.execute().actionGet();
	System.out.println("done");
} 
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE0CUreVbvpQrtXE186yqjA%2BtiYPCfZkwK_3QxfoOnOSA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE0CUreVbvpQrtXE186yqjA%2BtiYPCfZkwK_3QxfoOnOSA%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8F8C7196-83E8-48BC-9AF3-94F866A65E8B%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 7, 2015, 6:29pm

I understand both opinions, for give you a reference of my project, it is
based in migrate data from oracle to ES for analyze it with kibana,
specially are logs, millions of logs (around 90.000.000 of lines) and I'm
looking for the best way to do that, I don't know how many cluster I have
to use because is inserted data all days, I just to insert data like David
explain me and I can fix the error that I have and all worked very well
also I can see my data in Kibana. I have another question, how can I insert
not only 1000 registers like I'm doing now, else all my data that is
around 90.000.000, like I told you before and this increment so fast.

Thank you both guys.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30010282-64e5-4b94-9f9a-f16aa0a7659e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · January 7, 2015, 10:21pm

You can change the logic for "count < 1000", submit bulk request at each
count of 1000 with condition "count % 1000 == 0" or something, and iterate
with resultset.next() through all rows.

If you want to load rows with less stress on the Oracle read transactional
store and less memory consumption, you should not use a single SQL
statement but several ones to partition the SQL result set into smaller
ones, each fetching parts of the 90m rows.

Jörg

On Wed, Jan 7, 2015 at 7:29 PM, Marian Valero marianv0992@gmail.com wrote:

I understand both opinions, for give you a reference of my project, it is
based in migrate data from oracle to ES for analyze it with kibana,
specially are logs, millions of logs (around 90.000.000 of lines) and I'm
looking for the best way to do that, I don't know how many cluster I have
to use because is inserted data all days, I just to insert data like David
explain me and I can fix the error that I have and all worked very well
also I can see my data in Kibana. I have another question, how can I insert
not only 1000 registers like I'm doing now, else all my data that is
around 90.000.000, like I told you before and this increment so fast.

Thank you both guys.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/30010282-64e5-4b94-9f9a-f16aa0a7659e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/30010282-64e5-4b94-9f9a-f16aa0a7659e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEzH2MmRbL0Mt1LkLZ18PNCxjzvOGt8S2my%3DtwcsuoB6A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 8, 2015, 1:15pm

Ok! thank you, and referent to cluster, because is so many data an everyday
this increment and I can't have all data in only one machine, how many
cluster I have to use?

Thanks for all.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0886b197-b9e9-4cea-bce5-64f0a47b203d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · January 8, 2015, 10:37pm

You have to try capacity planning for yourself, by adjusting configuration
of index / shards / nodes and resource control parameters like heap size,
how many documents your system can load. There is no fixed rule that can
estimate the maximum amount of data volume an ES cluster is capable of.
Note that you can always add nodes and create new indices, it is possible
to grow a single cluster into hundreds or thousands of nodes ( = machines).

For a rough number, you should estimate your retention data volume, maybe
you must keep the data for some days and you can drop it then. Then you
have to estimate the resources for search (quer load, filter caches etc.)
All these factors should be taken into consideration for the number of
nodes you may need.

Jörg

On Thu, Jan 8, 2015 at 2:15 PM, Marian Valero marianv0992@gmail.com wrote:

Ok! thank you, and referent to cluster, because is so many data an
everyday this increment and I can't have all data in only one machine, how
many cluster I have to use?

Thanks for all.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0886b197-b9e9-4cea-bce5-64f0a47b203d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0886b197-b9e9-4cea-bce5-64f0a47b203d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHMBfCKmrjvgWRwwJP6SsHQqw9Q3qmz93azDWOGJWay4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ed_Kim · January 13, 2015, 8:52pm

Not sure if this is an option for you, but if an application is feeding
that log data into oracle, you could consider having that application also
index into Elasticsearch.

On Wednesday, January 7, 2015 at 5:46:18 AM UTC-8, Marian Valero wrote:

I'm reading data from my oracle database to java with JDBC but I want to
know how can I input the data that I'm getting in Elasticsearch in real
time.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marian_Valero · January 13, 2015, 9:07pm

Thanks Ed.

Yes, but I have data from 10 years ago and I have to input this data too
for analyze that logs with the data that is insert every day. There are
millions of logs.

2015-01-13 16:22 GMT-04:30 Ed Kim edkim81@gmail.com:

Not sure if this is an option for you, but if an application is feeding
that log data into oracle, you could consider having that application also
index into Elasticsearch.

On Wednesday, January 7, 2015 at 5:46:18 AM UTC-8, Marian Valero wrote:

I'm reading data from my oracle database to java with JDBC but I want to
know how can I input the data that I'm getting in Elasticsearch in real
time.

Thanks.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADT%2BR%3DZ6LNFP-0m%2BwHdwkj1MRupD2_Ybbbo7A9dSKtE126v%3DbA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Elasticsearch to Oracle Table Elasticsearch	11	756	November 11, 2020
How to ingest data in real time from oracle to Elasticsearch Elasticsearch	2	11703	October 5, 2017
Insert data into Elastic Search using Java Elasticsearch	9	14926	July 6, 2017
Elastic search connection with database in java Api Elasticsearch	2	326	July 6, 2017
Real time data ..flush? Elasticsearch	2	501	July 5, 2017

How can I input data from java to ES in real time?

Thanks.

Related topics