How to use output tuple values from tail pipe as parameters in Elasticsearch script?

Elastic Stack Elasticsearch
1

Hi all

I'm working with Cascading and ES-Hadoop connector, using an Hfs source and an ES sink on a Hadoop2MR1FlowConnector with following properties:

props.setProperty(ConfigurationOptions.ES_WRITE_OPERATION, ConfigurationOptions.ES_OPERATION_UPSERT);
props.setProperty(ConfigurationOptions.ES_UPDATE_SCRIPT, esUpdateScript);
props.setProperty(ConfigurationOptions.ES_UPDATE_SCRIPT_LANG, "groovy");

new Hadoop2MR1FlowConnector(props).connect(hfsTap, esTap, tailPipe).complete();

I see there is a config option to set script parameters: ConfigurationOptions.ES_UPDATE_SCRIPT_PARAMS_JSON

My question is, How can I use output tuple values (from tailPipe) as parameters in the groovy script?

Thanks

2

If you want to know more about the various configuration options you can:

  1. look at the official reference docs
  2. if looking at the sources, take a look at the test suite

Note that the params are static - they can either be constant values or extracted from the document passed to ES.
If your tuple is passed as the sink/tap to be written, then one can extract it from it - if not, then you'll have to device your own custom solution.

3

@costin thanks!