# Pb adding elements to array of tuples with pig on es-hadoop

**URL:** <https://discuss.elastic.co/t/pb-adding-elements-to-array-of-tuples-with-pig-on-es-hadoop/23421>\
**Category:** Elasticsearch\
**Created:** [April 27, 2015, 3:03pm UTC](https://discuss.elastic.co/t/pb-adding-elements-to-array-of-tuples-with-pig-on-es-hadoop/23421 "2015-04-27T15:03:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Philippe\_de\_Cuzey](https://avatars.discourse-cdn.com/v4/letter/p/919ad9/32.png) [@Philippe\_de\_Cuzey](https://discuss.elastic.co/u/Philippe_de_Cuzey)\
**Post date:** [April 27, 2015, 3:03pm UTC](https://discuss.elastic.co/t/pb-adding-elements-to-array-of-tuples-with-pig-on-es-hadoop/23421/1 "2015-04-27T15:03:56Z")

</div>

Hi all,

I have a PIG table hosting tomcat logs with 3 fields : log\_date, log\_url,  
log\_nb.  
I want to store this in ES with an index on log\_url, each document having  
an array of nested maps for each day :

{ "log\_url": "[http://www.xxx.fr/index.html](http://www.xxx.fr/index.html)",  
"log\_hits": [  
{  
"log\_nb": 1,  
"log\_date": "20150406"  
} ,  
{  
"log\_nb": 2,  
"log\_date": "20150407"  
}  
]  
}

This script will be run everyday, generating a new entry for each url. So  
for a given log\_url, the array will grow 1 element each day

as stated in the es-hadoop documentation  
[http://www.elastic.co/guide/en/elasticsearch/hadoop/current/pig.html#tuple-names](http://www.elastic.co/guide/en/elasticsearch/hadoop/current/pig.html#tuple-names),  
if we set es.mapping.pig.tuple.use.field.names (by default false) to true,  
tuples will be considered as array of maps when storing into ES.

The PIG code looks like :

b = LOAD ......  
c = group b BY log\_url;  
d = FOREACH c GENERATE  
group AS log\_url,  
TOTUPLE (log\_date, log\_nb) AS log\_hits;

store d into 'myindex/myindex'  
using org.elasticsearch.hadoop.pig.EsStorage (  
'es.mapping.pig.tuple.use.field.names=true',  
'es.write.operation=upsert',  
'es.mapping.id=log\_url',  
);

First time I launch it, it creates the following record :

{ "log\_url": "[http://www.xxx.fr/index.html](http://www.xxx.fr/index.html)",  
"log\_hits": [  
{  
"log\_nb": 1,  
"log\_date": "20150406"  
}  
]  
}

so far, so good,

when run again with a new date (say "20150407"), instead of inserting a new  
entry in the embedded array "log\_hits", it will replace its single array  
element and the ES document will become :

{ "log\_url": "[http://www.xxx.fr/index.html](http://www.xxx.fr/index.html)",  
"log\_hits": [  
{  
"log\_nb": 2,  
"log\_date": "20150407"  
}  
]  
}

I was expecting to get

{ "log\_url": "[http://www.xxx.fr/index.html](http://www.xxx.fr/index.html)",  
"log\_hits": [  
{  
"log\_nb": 1,  
"log\_date": "20150406"  
} ,  
{  
"log\_nb": 2,  
"log\_date": "20150407"  
}  
]  
}

Is there a way to achieve that ?

thanks

Philippe

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f62ee6fb-9efc-4fd8-82f7-a3b6cf594758%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f62ee6fb-9efc-4fd8-82f7-a3b6cf594758%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:17am UTC](https://discuss.elastic.co/t/pb-adding-elements-to-array-of-tuples-with-pig-on-es-hadoop/23421/2 "2017-07-06T00:17:35Z")

</div>

Closed on July 6, 2017, 12:17am UTC