Hello,
I'm trying ES-Hadoop to integrate it in our process. I'm using ES 2.1, Pig 0.14 and the version 2.2.0-rc1 of ES-Hadoop.
I need to write on multiple indexes/types at the time.
I'm reading a file with the current line :
{"partitionId":15,"siteId":"br2ryqd66","visitorId":"0001525cf5423a334e3df","visitId":"00015bbf7a52c4cbba536","eventId":"eawe38cukbpqfmuaursoszqjnly819fs","ts":"2016-01-07T23:54:24.824Z","eventType":"visit","eventName":"visit_closed","eventLive":1,"visit":{},"partner":{},"visit_closed":{},"meta":{"type":"event", "index":"v00000262"}}
I'm trying to store it on ES with Pig using EsStorage.
The command
STORE A INTO 'v00000262/event' USING org.elasticsearch.hadoop.pig.EsStorage('es.input.json=true','es.http.timeout = 5m', 'es.index.auto.create = false', 'es.mapping.id=eventId', 'es.mapping.timestamp=ts', 'es.mapping.parent=visitorId', 'es.mapping.exclude=meta','es.nodes=$es_url');
works perfectly but the command
STORE A INTO 'v00000262/{meta.type}' USING org.elasticsearch.hadoop.pig.EsStorage('es.input.json=true','es.http.timeout = 5m', 'es.index.auto.create = false', 'es.mapping.id=eventId', 'es.mapping.timestamp=ts', 'es.mapping.parent=visitorId', 'es.mapping.exclude=meta','es.nodes=$es_url');
returns the error Invalid target URI HEAD@null/v00000262/{meta.type}
In the log, I have (I've deleted the ES IP but it's the right one) :
================================================================================
Pig Stack Trace
ERROR 1002: Unable to store alias A
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias A
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1694)
at org.apache.pig.PigServer.registerQuery(PigServer.java:623)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1063)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:558)
at org.apache.pig.Main.main(Main.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[...:9200]]
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:383)
at org.elasticsearch.hadoop.rest.RestClient.executeNotFoundAllowed(RestClient.java:391)
at org.elasticsearch.hadoop.rest.RestClient.exists(RestClient.java:467)
at org.elasticsearch.hadoop.rest.RestRepository.indexExists(RestRepository.java:449)
at org.elasticsearch.hadoop.rest.InitializationUtils.checkIndexExistence(InitializationUtils.java:203)
at org.elasticsearch.hadoop.mr.EsOutputFormat.init(EsOutputFormat.java:263)
at org.elasticsearch.hadoop.mr.EsOutputFormat.checkOutputSpecs(EsOutputFormat.java:233)
at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1767)
at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1443)
at org.apache.pig.PigServer.execute(PigServer.java:1356)
at org.apache.pig.PigServer.access$500(PigServer.java:113)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1689)
... 14 more
I don't see where I've done a mistake...
Is it a bug or I've forgotten something ?
Thanks in advance