Full data in HDFS and Elasticsearch keeps the index pointer

Elastic Stack Elasticsearch
1

Hi all,

In order to solve the query/ aggregation issue, I want to park index data
in HDFS and ES keeps the index pointer.

Does anyone know how to do that? Please give me a guide, thx. I know
someone use MySQL to keep the index data, it's very like what I am talking
about.

However HDFS is more attractive to me, becasue HDFS is a centralize place
that I can share it to different hadoop ecosystem.

Best Regards,
Orozco

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7224ed04-4cc0-4aaf-a08f-038827023c4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

What is the query/aggregation issue that you are referring to?

On 9/22/14 12:14 PM, Hsu Orozco wrote:

Hi all,

In order to solve the query/ aggregation issue, I want to park index data in HDFS and ES keeps the index pointer.

Does anyone know how to do that? Please give me a guide, thx. I know someone use MySQL to keep the index data, it's very
like what I am talking about.

However HDFS is more attractive to me, becasue HDFS is a centralize place that I can share it to different hadoop ecosystem.

Best Regards,
Orozco

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7224ed04-4cc0-4aaf-a08f-038827023c4e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7224ed04-4cc0-4aaf-a08f-038827023c4e%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/541FE9CF.7030607%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Costin,

my query looks like this:

POST _search
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must_not": [
{
"terms": {
"host": ["","Null"]
}
},{
"terms": {
"curr_subscr_id": ["","Null"]
}
}
]
}
}
}
},
"aggs":{
"host_aggs":{
"terms":{
"field":"host"
},
"aggs":{
"subscr_aggs":{
"cardinality":{
"field":"curr_subscr_id"
}
}
}
}
}
,"size": 1
,"fields": [
"age","start_time","host","_source"
]
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb3b27b0-9ff2-435a-a216-f76079541dc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

And another one thing,

I was misunderstanding, ES can't parallel work with another repository such
like MySQL. I mean ES can't keep index pointer in itself, and index data in
MySQL.
All searching process must be done in ES.

The only way improving performane seems like adding more machines, so far I
think.

Hsu Orozco於 2014年9月22日星期一UTC+8下午5時14分56秒寫道：

Hi all,

In order to solve the query/ aggregation issue, I want to park index data
in HDFS and ES keeps the index pointer.

Does anyone know how to do that? Please give me a guide, thx. I know
someone use MySQL to keep the index data, it's very like what I am talking
about.

However HDFS is more attractive to me, becasue HDFS is a centralize place
that I can share it to different hadoop ecosystem.

Best Regards,
Orozco

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aed46066-bf80-4f21-a329-d5779734f316%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.