# Using Pig/Spark on ElasticSearch (as External Storage)
**URL:** https://discuss.elastic.co/t/using-pig-spark-on-elasticsearch-as-external-storage/20205
**Category:** Elasticsearch
**Created:** [October 12, 2014, 12:51pm UTC](https://discuss.elastic.co/t/using-pig-spark-on-elasticsearch-as-external-storage/20205 "2014-10-12T12:51:11Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@zkidkid](https://discuss.elastic.co/u/zkidkid)
#### Post date: [October 12, 2014, 12:51pm UTC](https://discuss.elastic.co/t/using-pig-spark-on-elasticsearch-as-external-storage/20205/1 "2014-10-12T12:51:11Z")
Hi All,
Currently I am using ElasticSearch for a logging system.
My first solution is that every log will put on ES and index will rolling
by date.
To do real time stats, I will use Aggregation.
To do statistic I will use Spark (or Hive, Shark whatever) on ES data
(thanks to ElasticSearch-Hadoop plugin
All is fine, but when my data grows (currently 17M record/index/date),
Spark (Hive also) becomes very slow.
I did benmark with the same data on ES and Hadoop, and I saw that Spark
(Hive) run on Hadoop is much faster.
Is there something I have missed ?
Currently I maintain two storage, one is ES for realtime stats, another is
Hadoop for other statistics. Is it fine ?
I really appreciate any idea or solution.,
Thanks.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [October 12, 2014, 1:02pm UTC](https://discuss.elastic.co/t/using-pig-spark-on-elasticsearch-as-external-storage/20205/2 "2014-10-12T13:02:49Z")
It depends on various factors. Do you put all the data under one index or
is it one index per day/month/hour? What type of script and performance
degradation do you see? If it's easier feel free to reach out on irc. I'll
be traveling this week but we'll be back the next one.
Cheers
On Oct 12, 2014 2:51 PM, "Sang Dang" [zkidkid@gmail.com](mailto:zkidkid@gmail.com) wrote:
> Hi All,
> Currently I am using Elasticsearch for a logging system.
> My first solution is that every log will put on ES and index will rolling
> by date.
> To do real time stats, I will use Aggregation.
> To do statistic I will use Spark (or Hive, Shark whatever) on ES data
> (thanks to Elasticsearch-Hadoop plugin
> All is fine, but when my data grows (currently 17M record/index/date),
> Spark (Hive also) becomes very slow.
>
> I did benmark with the same data on ES and Hadoop, and I saw that Spark
> (Hive) run on Hadoop is much faster.
>
> Is there something I have missed ?
>
> Currently I maintain two storage, one is ES for realtime stats, another is
> Hadoop for other statistics. Is it fine ?
>
> I really appreciate any idea or solution.,
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdme%2BnCuz1tkmCtGX7Amq1%3Db2%3DirBwBVJxk\_nGySja\_PuoQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdme%2BnCuz1tkmCtGX7Amq1%3Db2%3DirBwBVJxk_nGySja_PuoQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@zkidkid](https://discuss.elastic.co/u/zkidkid)
#### Post date: [October 12, 2014, 1:41pm UTC](https://discuss.elastic.co/t/using-pig-spark-on-elasticsearch-as-external-storage/20205/3 "2014-10-12T13:41:30Z")
Hi Costin Leau,
Currently I just pull all data in one index (INDEX\_NAME\_DATE)
In my benmark, I just do two function, count and count distinct field.
P/S: Thanks for your fast response, I would really happy to see you at IRC
(just give me the time).
On Sunday, October 12, 2014 8:02:57 PM UTC+7, Costin Leau wrote:
> It depends on various factors. Do you put all the data under one index or
> is it one index per day/month/hour? What type of script and performance
> degradation do you see? If it's easier feel free to reach out on irc. I'll
> be traveling this week but we'll be back the next one.
> Cheers
> On Oct 12, 2014 2:51 PM, "Sang Dang" \<[zki...@gmail.com](mailto:zki...@gmail.com) \\>
> wrote:
>
> > Hi All,
> > Currently I am using Elasticsearch for a logging system.
> > My first solution is that every log will put on ES and index will rolling
> > by date.
> > To do real time stats, I will use Aggregation.
> > To do statistic I will use Spark (or Hive, Shark whatever) on ES data
> > (thanks to Elasticsearch-Hadoop plugin
> > All is fine, but when my data grows (currently 17M record/index/date),
> > Spark (Hive also) becomes very slow.
> >
> > I did benmark with the same data on ES and Hadoop, and I saw that Spark
> > (Hive) run on Hadoop is much faster.
> >
> > Is there something I have missed ?
> >
> > Currently I maintain two storage, one is ES for realtime stats, another
> > is Hadoop for other statistics. Is it fine ?
> >
> > I really appreciate any idea or solution.,
> >
> > Thanks.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/efc5ac1e-863c-49ee-b13a-211fb03a54c6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/efc5ac1e-863c-49ee-b13a-211fb03a54c6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:56am UTC](https://discuss.elastic.co/t/using-pig-spark-on-elasticsearch-as-external-storage/20205/4 "2017-07-06T00:56:40Z")