# Some Bulk Indexing PRoblems

**URL:** <https://discuss.elastic.co/t/some-bulk-indexing-problems/19685>\
**Category:** Elasticsearch\
**Created:** [September 9, 2014, 2:24pm UTC](https://discuss.elastic.co/t/some-bulk-indexing-problems/19685 "2014-09-09T14:24:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joshua\_P](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@Joshua\_P](https://discuss.elastic.co/u/Joshua_P)\
**Post date:** [September 9, 2014, 2:24pm UTC](https://discuss.elastic.co/t/some-bulk-indexing-problems/19685/1 "2014-09-09T14:24:13Z")

</div>

Hi there! Sorry I posted two topics. I've somehow managed to post an  
incomplete post.

I'm trying to do a one-time index of about 800,000 records into an instance  
of elasticsearch. But I'm having a bit of trouble. It continually fails  
around 200,000 records. Looking at in the Elasticsearch Head Plugin, my  
index goes offline and becomes unrecoverable.

For now, I have it running on a VM on my personal machine.

VM Config:  
Ubuntu Server 14.04 64-Bit  
8 GB RAM  
2 Processors  
32 GB SSD

Java  
java version "1.7.0\_65"  
OpenJDK Runtime Environment (IcedTea 2.5.1) (7u65-2.5.1-4ubuntu1~0.14.04.  
2)  
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

Elasticsearch is using mostly the defaults. This is the output of:  
curl [http://localhost:9200/\_nodes/process?pretty](http://localhost:9200/_nodes/process?pretty)  
{  
"cluster\_name" : "property\_transaction\_data",  
"nodes" : {  
"KlFkO\_qgSOKmV\_jjj5xeVw" : {  
"name" : "Marvin Flumm",  
"transport\_address" : "inet[/192.168.133.131:9300]",  
"host" : "ubuntu-es",  
"ip" : "127.0.1.1",  
"version" : "1.3.2",  
"build" : "dee175d",  
"http\_address" : "inet[/192.168.133.131:9200]",  
"process" : {  
"refresh\_interval\_in\_millis" : 1000,  
"id" : 1092,  
"max\_file\_descriptors" : 65535,  
"mlockall" : true  
}  
}  
}  
}

I adjusted ES\_HEAP\_SIZE to 512mb.

I'm using the following code to pull data from SQL Server and index it. Can  
someone give me a hand and tell me what I might be doing wrong?

In the code below, PropertyGeneralInfoRow is just a class to represent a  
row coming from the SQL view.

package com.rca.index;

import com.fasterxml.jackson.databind.ObjectMapper;  
import com.rca.database.DBConnection;  
import com.rca.database.PropertyGeneralInfoRow;  
import org.apache.logging.log4j.Logger;  
import org.elasticsearch.action.ActionRequest;  
import org.elasticsearch.action.bulk.\*;  
import org.elasticsearch.action.index.IndexRequest;  
import org.elasticsearch.client.Client;  
import org.elasticsearch.client.transport.TransportClient;  
import org.elasticsearch.common.settings.ImmutableSettings;  
import org.elasticsearch.common.settings.Settings;  
import org.elasticsearch.common.transport.InetSocketTransportAddress;

import java.io.IOException;  
import java.util.List;  
import org.apache.logging.log4j.LogManager;

/\*\*

- Created by jpetersen on 8/1/14.  
\*/  
public class Indexer {

WHERE Country\_id = 1";

```
    System.out.println("getting data");
    List<PropertyGeneralInfoRow> pgiTable = dbConn.

```

ExecuteQueryWithoutParameters(query);  
System.out.println("got data");

```
    ObjectMapper mapper = new ObjectMapper();

    Settings settings = ImmutableSettings.settingsBuilder().put(

```

"cluster.name", "property\_transaction\_data").build();

```
    Client client = new TransportClient(settings).addTransportAddress(

```

new InetSocketTransportAddress("192.168.133.131", 9300));

```
    BulkProcessor bulkProcessor = BulkProcessor.builder(client, new 

```

BulkProcessor.Listener() {  
@Override  
public void beforeBulk(long executionId, BulkRequest request) {  
System.out.println("about to index " + request.  
numberOfActions() + " records.");  
}

```
        @Override
        public void afterBulk(long executionId, BulkRequest request, 

```

BulkResponse response) {  
System.out.println("successfully indexed " + request.  
numberOfActions() + " records in " + response.getTook() + ".");  
}

```
        @Override
        public void afterBulk(long executionId, BulkRequest request, 

```

Throwable failure) {  
System.out.println("failure somewhere on " + request.  
toString());  
failure.printStackTrace();  
logger.warn("failure on " + request.toString());  
}  
}).setBulkActions(500).setConcurrentRequests(1).build();

```
    for( int i = 0; i < pgiTable.size(); i++ ){
        //prep location field
        PropertyGeneralInfoRow pgiRow = pgiTable.get(i);

        Double[] location = {pgiRow.getLon_dbl(), pgiRow.getLat_dbl()};

        geocode geocode = new geocode();

        geocode.setLocation(location);

        pgiRow.setGeocode(geocode);

        // prep full address string
        pgiRow.setFulladdressstring(pgiRow.getPropertykey_tx() + ", " +
                pgiRow.getCity_tx() + ", " + pgiRow.getStateprov_cd() +
                ", " + pgiRow.getCountry_tx() + ", " + pgiRow.

```

getPostalcode\_tx());

```
        String jsonRow = mapper.writeValueAsString(pgiRow);

        if( jsonRow != null && !jsonRow.isEmpty() && !jsonRow.equals(

```

"{}") ){  
bulkProcessor.add(new IndexRequest("rcapropertydata",  
"rcaproperty").source(jsonRow.getBytes()));  
// bulkProcessor.add(client.prepareIndex("rcapropertydata",  
"rcaproperty").setSource(jsonRow));  
}  
else{  
// don't add null strings..  
try{  
System.out.println(pgiRow.toString());  
}  
catch (Exception e){  
System.out.println("Some error in toString() ...");  
}  
System.out.println("Some json output was null. -- " + pgiRow  
.getProperty\_id().toString());  
}

```
    }

    bulkProcessor.flush();
    bulkProcessor.close();

}

```

}

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/864372f1-7a1b-4f9f-aac0-87688b60ccd3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/864372f1-7a1b-4f9f-aac0-87688b60ccd3%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\
**Post date:** [September 9, 2014, 2:54pm UTC](https://discuss.elastic.co/t/some-bulk-indexing-problems/19685/2 "2014-09-09T14:54:00Z")

</div>

Hello Joshua ,

Please refrain from posting the same question twice.  
If you need to add additional information , just reply to the original  
thread.

Thanks  
Vineeth

On Tue, Sep 9, 2014 at 7:54 PM, Joshua P [jpetersen841@gmail.com](mailto:jpetersen841@gmail.com) wrote:

> Hi there! Sorry I posted two topics. I've somehow managed to post an  
> incomplete post.
> 
> I'm trying to do a one-time index of about 800,000 records into an  
> instance of elasticsearch. But I'm having a bit of trouble. It continually  
> fails around 200,000 records. Looking at in the Elasticsearch Head Plugin,  
> my index goes offline and becomes unrecoverable.
> 
> For now, I have it running on a VM on my personal machine.
> 
> VM Config:  
> Ubuntu Server 14.04 64-Bit  
> 8 GB RAM  
> 2 Processors  
> 32 GB SSD
> 
> Java  
> java version "1.7.0\_65"  
> OpenJDK Runtime Environment (IcedTea 2.5.1) (7u65-2.5.1-4ubuntu1~0.14.04.  
> 2)  
> OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
> 
> Elasticsearch is using mostly the defaults. This is the output of:  
> curl [http://localhost:9200/\_nodes/process?pretty](http://localhost:9200/_nodes/process?pretty)  
> {  
> "cluster\_name" : "property\_transaction\_data",  
> "nodes" : {  
> "KlFkO\_qgSOKmV\_jjj5xeVw" : {  
> "name" : "Marvin Flumm",  
> "transport\_address" : "inet[/192.168.133.131:9300]",  
> "host" : "ubuntu-es",  
> "ip" : "127.0.1.1",  
> "version" : "1.3.2",  
> "build" : "dee175d",  
> "http\_address" : "inet[/192.168.133.131:9200]",  
> "process" : {  
> "refresh\_interval\_in\_millis" : 1000,  
> "id" : 1092,  
> "max\_file\_descriptors" : 65535,  
> "mlockall" : true  
> }  
> }  
> }  
> }
> 
> I adjusted ES\_HEAP\_SIZE to 512mb.
> 
> I'm using the following code to pull data from SQL Server and index it.  
> Can someone give me a hand and tell me what I might be doing wrong?
> 
> In the code below, PropertyGeneralInfoRow is just a class to represent a  
> row coming from the SQL view.
> 
> package com.rca.index;
> 
> import com.fasterxml.jackson.databind.ObjectMapper;  
> import com.rca.database.DBConnection;  
> import com.rca.database.PropertyGeneralInfoRow;  
> import org.apache.logging.log4j.Logger;  
> import org.elasticsearch.action.ActionRequest;  
> import org.elasticsearch.action.bulk.\*;  
> import org.elasticsearch.action.index.IndexRequest;  
> import org.elasticsearch.client.Client;  
> import org.elasticsearch.client.transport.TransportClient;  
> import org.elasticsearch.common.settings.ImmutableSettings;  
> import org.elasticsearch.common.settings.Settings;  
> import org.elasticsearch.common.transport.InetSocketTransportAddress;
> 
> import java.io.IOException;  
> import java.util.List;  
> import org.apache.logging.log4j.LogManager;
> 
> /\*\*
> 
> - Created by jpetersen on 8/1/14.  
> \*/  
> public class Indexer {
> 
> WHERE Country\_id = 1";
> 
> ```
> System.out.println("getting data");
> List<PropertyGeneralInfoRow> pgiTable = dbConn.
> 
> ```
> 
> ExecuteQueryWithoutParameters(query);  
> System.out.println("got data");
> 
> ```
> ObjectMapper mapper = new ObjectMapper();
> 
> Settings settings = ImmutableSettings.settingsBuilder().put("
> 
> ```
> 
> cluster.name", "property\_transaction\_data").build();
> 
> ```
> Client client = new TransportClient(settings).addTransportAddress(
> 
> ```
> 
> new InetSocketTransportAddress("192.168.133.131", 9300));
> 
> ```
> BulkProcessor bulkProcessor = BulkProcessor.builder(client, new
> 
> ```
> 
> BulkProcessor.Listener() {  
> @Override  
> public void beforeBulk(long executionId, BulkRequest request)  
> {  
> System.out.println("about to index " + request.  
> numberOfActions() + " records.");  
> }
> 
> ```
> @Override
> public void afterBulk(long executionId, BulkRequest request,
> 
> ```
> 
> BulkResponse response) {  
> System.out.println("successfully indexed " + request.  
> numberOfActions() + " records in " + response.getTook() + ".");  
> }
> 
> ```
> @Override
> public void afterBulk(long executionId, BulkRequest request,
> 
> ```
> 
> Throwable failure) {  
> System.out.println("failure somewhere on " + request.  
> toString());  
> failure.printStackTrace();  
> logger.warn("failure on " + request.toString());  
> }  
> }).setBulkActions(500).setConcurrentRequests(1).build();
> 
> ```
> for( int i = 0; i < pgiTable.size(); i++ ){
> //prep location field
> PropertyGeneralInfoRow pgiRow = pgiTable.get(i);
> 
> Double[] location = {pgiRow.getLon_dbl(), pgiRow.getLat_dbl
> 
> ```
> 
> ()};
> 
> ```
> geocode geocode = new geocode();
> 
> geocode.setLocation(location);
> 
> pgiRow.setGeocode(geocode);
> 
> // prep full address string
> pgiRow.setFulladdressstring(pgiRow.getPropertykey_tx() + ", "
> 
> ```
> 
> - 
> 
> ```
> pgiRow.getCity_tx() + ", " + pgiRow.getStateprov_cd()
> 
> ```
> 
> - 
> 
> ```
> ", " + pgiRow.getCountry_tx() + ", " + pgiRow.
> 
> ```
> 
> getPostalcode\_tx());
> 
> ```
> String jsonRow = mapper.writeValueAsString(pgiRow);
> 
> if( jsonRow != null && !jsonRow.isEmpty() && !jsonRow.equals(
> 
> ```
> 
> "{}") ){  
> bulkProcessor.add(new IndexRequest("rcapropertydata",  
> "rcaproperty").source(jsonRow.getBytes()));  
> //  
> bulkProcessor.add(client.prepareIndex("rcapropertydata",  
> "rcaproperty").setSource(jsonRow));  
> }  
> else{  
> // don't add null strings..  
> try{  
> System.out.println(pgiRow.toString());  
> }  
> catch (Exception e){  
> System.out.println("Some error in toString() ...");  
> }  
> System.out.println("Some json output was null. -- " +  
> pgiRow.getProperty\_id().toString());  
> }
> 
> ```
> }
> 
> bulkProcessor.flush();
> bulkProcessor.close();
> 
> }
> 
> ```
> 
> }
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/864372f1-7a1b-4f9f-aac0-87688b60ccd3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/864372f1-7a1b-4f9f-aac0-87688b60ccd3%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/864372f1-7a1b-4f9f-aac0-87688b60ccd3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/864372f1-7a1b-4f9f-aac0-87688b60ccd3%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5mVRwOB\_tyZjgH%2B%3DmG22OVfx%2BqeJHP%2BkG291qOL7%3Dsxhw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5mVRwOB_tyZjgH%2B%3DmG22OVfx%2BqeJHP%2BkG291qOL7%3Dsxhw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:03am UTC](https://discuss.elastic.co/t/some-bulk-indexing-problems/19685/3 "2017-07-06T01:03:37Z")

</div>