Equivalent of setCreate() for a particular document

[jnioche]

I am trying to do the equivalent of

 IndexRequestBuilder request = connection.getClient()
    .prepareIndex(indexName, docType).setSource(builder)
    .setCreate(create).setId(sha256hex);

    if (StringUtils.isNotBlank(partitionKey)) {
        request.setRouting(partitionKey);
    }

with the Storm module of Elasticsearch-Hadoop.

I've worked out how to set the index name and type and the write operation for the whole index via es.write.operation but would like to override the latter for specific documents as illustrated in the example above. I could not find a way of doing this in the docs.

Also I found how to declare the doc id via

 settings.setProperty("es.mapping.id", "sha256");

but ideally I'd rather not add that field to the source, same for the field used for sharding. Is that possible?

Thanks

[james.baiera]

@jnioche due to the nature of the connector only functioning on documents and a configuration, anything that modifies the routing of a document on a document-by-document-basis must be included in the document information for the connector to extract it.

Of course, this means that those "meta" fields that were only meant to modify where the document is routed or what the document id is end up in the document itself. To keep those fields from polluting the final document sent to Elasticsearch, you can blacklist these "meta" fields from being using in the final serialization by setting them as a comma separated list in the es.mapping.exclude property.

[jnioche]

Thanks for the explanation @james.baiera. For the first part of my question, is there a way I can set the value of 'create' on a per document basis?

[james.baiera]

@jnioche At the moment we do not support dynamic operation switching based on document content. Perhaps you could use some sort of scripting to work around this at the moment?

[jnioche]

that's good to know, thanks @james.baiera