# Is it possible to write to ES from a json file in HDFS where JSON file has inconsistent or different keys in different records **URL:** \ **Category:** Elasticsearch\ **Created:** [April 1, 2014, 6:14pm UTC](https://discuss.elastic.co/t/is-it-possible-to-write-to-es-from-a-json-file-in-hdfs-where-json-file-has-inconsistent-or-different-keys-in-different-records/16744 "2014-04-01T18:14:40Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![siva\_mannem](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@siva\_mannem](https://discuss.elastic.co/u/siva_mannem)\ **Post date:** [April 1, 2014, 6:14pm UTC](https://discuss.elastic.co/t/is-it-possible-to-write-to-es-from-a-json-file-in-hdfs-where-json-file-has-inconsistent-or-different-keys-in-different-records/16744/1 "2014-04-01T18:14:40Z")
Hi, my json file is like this +++++++++++++++++++ {"k1":"v1" , "k2":"v2" , "k3":"v3" , "k4":"v4" , "k5":"v5"} {"k12":"v11" , "k23":"v22" , "k34":"v33" , "k45":"v44" , "k56":"v55"} {"k1":"v111" , "k2":"v222" , "k3":"v333" , "k4":"v444" , "k5":"v555"} {"k123":"v12" , "k234":"v23" , "k345":"v34" , "k456":"v45" , "k567":"v56"} +++++++++++++++++++++ my pig script is like this +++++++++++++++++++++++++++ REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar; DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.nodes=gateway1 , es.resource=ca/sf'); A = LOAD '/elastic\_search/in\_dir/' using JsonLoader('k1:chararray,k2:chararray,k3:chararray,k4:chararray,k5:chararray'); _B = FOREACH A GENERATE k1, k3, k5;_ +++++++++++++++++++++++++++++ I am expecting a output like this +++++++++++++++ (v1,v3,v5) (v111,v333,v555) ++++++++++++++++++ but i am getting a output like this ++++++++++++ (v1,v3,v5) _(v11,v33,v55)_ (v111,v333,v555) ++++++++++++++ is there any way to ignore the second record as there are no keys K1, K3 and k5 in second record? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/410f55ac-7e43-4789-83dc-eb4958fa2d55%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/410f55ac-7e43-4789-83dc-eb4958fa2d55%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![siva\_mannem](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@siva\_mannem](https://discuss.elastic.co/u/siva_mannem)\ **Post date:** [April 3, 2014, 4:45pm UTC](https://discuss.elastic.co/t/is-it-possible-to-write-to-es-from-a-json-file-in-hdfs-where-json-file-has-inconsistent-or-different-keys-in-different-records/16744/2 "2014-04-03T16:45:29Z")
sorry. i am getting a output like this ++++++++++++ (v1,v3,v5) _(v11,v33,v55)_ (v111,v333,v555) _(v12, v34, v56)_ ++++++++++++++ On Tuesday, April 1, 2014 11:14:40 AM UTC-7, siva mannem wrote: > Hi, > > my json file is like this > +++++++++++++++++++ > {"k1":"v1" , "k2":"v2" , "k3":"v3" , "k4":"v4" , > "k5":"v5"} > > {"k12":"v11" , "k23":"v22" , "k34":"v33" , "k45":"v44" , > "k56":"v55"} > > {"k1":"v111" , "k2":"v222" , "k3":"v333" , "k4":"v444" , "k5":"v555"} > > {"k123":"v12" , "k234":"v23" , "k345":"v34" , "k456":"v45" , > "k567":"v56"} > +++++++++++++++++++++ > > my pig script is like this > +++++++++++++++++++++++++++ > REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar; > > DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.nodes=gateway1 , > es.resource=ca/sf'); > > A = LOAD '/elastic\_search/in\_dir/' using > JsonLoader('k1:chararray,k2:chararray,k3:chararray,k4:chararray,k5:chararray'); > > _B = FOREACH A GENERATE k1, k3, k5;_ > +++++++++++++++++++++++++++++ > > I am expecting a output like this > +++++++++++++++ > (v1,v3,v5) > (v111,v333,v555) > ++++++++++++++++++ > > but i am getting a output like this > ++++++++++++ > (v1,v3,v5) > _(v11,v33,v55)_ > (v111,v333,v555) > ++++++++++++++ > > is there any way to ignore the second record as there are no keys K1, K3 > and k5 in second record? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/51fceb7f-fc25-410b-911c-f283f8ecf5e5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51fceb7f-fc25-410b-911c-f283f8ecf5e5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:38am UTC](https://discuss.elastic.co/t/is-it-possible-to-write-to-es-from-a-json-file-in-hdfs-where-json-file-has-inconsistent-or-different-keys-in-different-records/16744/3 "2017-07-06T01:38:31Z")
Closed on July 6, 2017, 1:38am UTC