Need Advice on combining similar types from different indexes

Elastic Stack Elasticsearch
#1

I am considering using Elastic search to create an index of information
about devices in the environment from different sources, and need some
advice on the best way to solve my problem or if elastic search is even a
good fit.
The goal is to index the important pieces of information we know about
devices (servers, switches etc.) from all the different sources in our
enterprise, with one of the key requirements being the ability to easily
see where discrepancies are.
The first way I was considering was to create an index per source and a
type per device type, A good example may help:
(these are just examples not actual statements that were run)
Getting the server from the different source indexes might yield something
similar to the two statements below:

curl -XGET 'http://127.0.0.1:9200/activeDirectory/server/servername1’
{
"_index":"activeDirectory",
"type":"server",
"id":"servername1",
"env":"prod",
"description" : "A super cool web server",
"etc":"lots of other fields"
}
curl -XGET 'http://127.0.0.1:9200/inventoryDB/server/servername1’
{
"_index":"inventoryDB",
"type":"server",
"id":"servername1",
"env":"test",
"owner":"Bob",
"etc":"other fields and values"
}

And then getting the server from the combined index would yield something
similar to:
curl -XGET 'http://127.0.0.1:9200/all/server/servername1’
{
"_index":"all",
"type":"server",
"id":"servername1",
"env":{ "activeDirectory" : "prod", "inventoryDB" : "test" },
"owner": {"inventoryDB" : "Bob"},
"description" : {"activeDirectory":"A super cool web server"},
"etc":{"activeDirectory" : "lots of other fields", inventoryDB " : "other
fields and values"}
}

I like the idea of having all sources in their own index, but that is not
as important as having the combined index so maybe it would be simpler to
just index the data as it comes in into the combined index. If I did that
could I use a custom mapping or is there a better way?

Thanks
Dale

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

#2

Hi Dale,
not sure what you mean by custom mapping but if you just intend to index
the same documents on different indices, you might want to have a look at
aliaseshttp://www.elasticsearch.org/guide/reference/api/admin-indices-aliases/,
which allows to have virtual indices you can refer to that point to one or
more physical index. An alias can also hold filters or a custom routing
value.

Hope this helps
Luca

On Friday, August 2, 2013 7:05:28 PM UTC+2, Dale Cameron wrote:

I am considering using Elastic search to create an index of information
about devices in the environment from different sources, and need some
advice on the best way to solve my problem or if elastic search is even a
good fit.
The goal is to index the important pieces of information we know about
devices (servers, switches etc.) from all the different sources in our
enterprise, with one of the key requirements being the ability to easily
see where discrepancies are.
The first way I was considering was to create an index per source and a
type per device type, A good example may help:
(these are just examples not actual statements that were run)
Getting the server from the different source indexes might yield something
similar to the two statements below:

curl -XGET 'http://127.0.0.1:9200/activeDirectory/server/servername1’
{
"_index":"activeDirectory",
"type":"server",
"id":"servername1",
"env":"prod",
"description" : "A super cool web server",
"etc":"lots of other fields"
}
curl -XGET 'http://127.0.0.1:9200/inventoryDB/server/servername1’
{
"_index":"inventoryDB",
"type":"server",
"id":"servername1",
"env":"test",
"owner":"Bob",
"etc":"other fields and values"
}

And then getting the server from the combined index would yield something
similar to:
curl -XGET 'http://127.0.0.1:9200/all/server/servername1’
{
"_index":"all",
"type":"server",
"id":"servername1",
"env":{ "activeDirectory" : "prod", "inventoryDB" : "test" },
"owner": {"inventoryDB" : "Bob"},
"description" : {"activeDirectory":"A super cool web server"},
"etc":{"activeDirectory" : "lots of other fields", inventoryDB " : "other
fields and values"}
}

I like the idea of having all sources in their own index, but that is not
as important as having the combined index so maybe it would be simpler to
just index the data as it comes in into the combined index. If I did that
could I use a custom mapping or is there a better way?

Thanks
Dale

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

#3