Documents not deleted when using DeleteRequest within BulkProcessor


#1

Trying to index/delete documents within one BulkProcessor object in the
Java API. Indexing documents works great! Deleting, however, does not.

bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId));

Nothing happens. Any ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

Do you try to delete a doc using an alias?
Any failure or error reported by the bulk processor?

Hard to tell more without seeing the code / data.

David

Le 30 avr. 2015 à 02:03, Diana Tuck dtuck9@gmail.com a écrit :

Trying to index/delete documents within one BulkProcessor object in the Java API. Indexing documents works great! Deleting, however, does not.

bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId));

Nothing happens. Any ideas?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/512A5411-A00C-4F02-B11E-A0B24A27E7E3%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


#3

Thank you for the reply, David.

We are using an alias to delete. Is that a problem? Indexing with the
alias through the bulk processor works fine.

There are no errors reported, it just seems to disappear into the oblivion.
Here's our code for the BulkProcessor:

public static BulkProcessor getBulkProcessor(Client client, int esConcurrencyLevel, int esBulkSize, int esFlushInterval) {
return BulkProcessor.builder(client, new BulkProcessor.Listener() {

    @Override
    public void beforeBulk(long executionId, BulkRequest bulkRequest) {
        LOG.debug("Processing {} requests in bulk process {}", bulkRequest.numberOfActions(), executionId);
    }

    @Override
    public void afterBulk(long executionId, BulkRequest bulkRequest, BulkResponse response) {

        if (response.hasFailures()) {
            for (BulkItemResponse item : response.getItems()) {
                LOG.error("Processing to index \"{}\" failed for entity id {} with message {}", item.getIndex(),
                        item.getId(), item.getFailureMessage());
            }
        }
    }

    @Override
    public void afterBulk(long executionId, BulkRequest bulkRequest, Throwable throwable) {
        LOG.error("Failed to process {} requests in bulk request {}: {}", bulkRequest.numberOfActions(),
                executionId, throwable.getMessage());
        throwable.printStackTrace();
    }
})
        .setBulkActions(esBulkSize)
        .setFlushInterval(TimeValue.timeValueSeconds(esFlushInterval))
        .setConcurrentRequests(esConcurrencyLevel)
        .build();

}

Code for the delete request:

bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId));

where index.getIndexingAlias() is an alias (same alias used for indexing which is working), type is the document type "company" and entityId is the document ID.

What data would be helpful? An example document, the index metadata, something else?

On Wednesday, April 29, 2015 at 9:53:41 PM UTC-7, David Pilato wrote:

Do you try to delete a doc using an alias?
Any failure or error reported by the bulk processor?

Hard to tell more without seeing the code / data.

David

Le 30 avr. 2015 à 02:03, Diana Tuck <dtu...@gmail.com <javascript:>> a
écrit :

Trying to index/delete documents within one BulkProcessor object in the
Java API. Indexing documents works great! Deleting, however, does not.

bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId));

Nothing happens. Any ideas?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/01b6ab18-78a8-44d0-b574-c649501ec21a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #4

No as soon as you have only one index for this alias, indexing and deleting should work.

I don’t see anything suspicious here.

Any chance you could share on github your full code?

When you say that "nothing happens", do you mean that you never get the debug LOG « Processing {} … » ?
Or do you mean that the document has not been removed ?

How do you test all that?

--
David Pilato - Developer | Evangelist
elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 30 avr. 2015 à 19:10, Diana Tuck dtuck9@gmail.com a écrit :

Thank you for the reply, David.

We are using an alias to delete. Is that a problem? Indexing with the alias through the bulk processor works fine.

There are no errors reported, it just seems to disappear into the oblivion. Here's our code for the BulkProcessor:

public static BulkProcessor getBulkProcessor(Client client, int esConcurrencyLevel, int esBulkSize, int esFlushInterval) {
return BulkProcessor.builder(client, new BulkProcessor.Listener() {

    @Override
    public void beforeBulk(long executionId, BulkRequest bulkRequest) {
        LOG.debug("Processing {} requests in bulk process {}", bulkRequest.numberOfActions(), executionId);
    }

    @Override
    public void afterBulk(long executionId, BulkRequest bulkRequest, BulkResponse response) {

        if (response.hasFailures()) {
            for (BulkItemResponse item : response.getItems()) {
                LOG.error("Processing to index \"{}\" failed for entity id {} with message {}", item.getIndex(),
                        item.getId(), item.getFailureMessage());
            }
        }
    }

    @Override
    public void afterBulk(long executionId, BulkRequest bulkRequest, Throwable throwable) {
        LOG.error("Failed to process {} requests in bulk request {}: {}", bulkRequest.numberOfActions(),
                executionId, throwable.getMessage());
        throwable.printStackTrace();
    }
})
        .setBulkActions(esBulkSize)
        .setFlushInterval(TimeValue.timeValueSeconds(esFlushInterval))
        .setConcurrentRequests(esConcurrencyLevel)
        .build();

}

Code for the delete request:

bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId));

where index.getIndexingAlias() is an alias (same alias used for indexing which is working), type is the document type "company" and entityId is the document ID.

What data would be helpful? An example document, the index metadata, something else?

On Wednesday, April 29, 2015 at 9:53:41 PM UTC-7, David Pilato wrote:
Do you try to delete a doc using an alias?
Any failure or error reported by the bulk processor?

Hard to tell more without seeing the code / data.

David

Le 30 avr. 2015 à 02:03, Diana Tuck <dtu...@gmail.com <javascript:>> a écrit :

Trying to index/delete documents within one BulkProcessor object in the Java API. Indexing documents works great! Deleting, however, does not.

bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId));

Nothing happens. Any ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/01b6ab18-78a8-44d0-b574-c649501ec21a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/01b6ab18-78a8-44d0-b574-c649501ec21a%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C7602498-6322-4F06-86ED-01B5147FA1FE%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Shakun Vohra) #5

Were you able to get this issue fixed? I'm having the same exact issue with BulkProcessor and delete and nothing is happening, no errors, no delete


#6

Unfortunately I cannot remember what I ended up doing, and I no longer have access to the code to verify. I vaguely remember having to use the actual index name as opposed to the alias.

For our use case, however, deletions were relatively infrequent, so an acceptable approach was to just submit the DeleteRequest individually as they came in via a Kafka message.

Sorry I couldn't be of more help.


(Shakun Vohra) #7

i had to do extra logging in "after bulk" and saw the errors :slight_smile:


(system) #8