Example config file

Elastic Stack Elasticsearch
1

Hiya

Does anybody have a real life example of an elasticsearch config file.

I'm somewhat confused about what goes into the config file, and what
gets specified when (eg) creating an index.

thanks

Clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

2

HI,

If you can give an example of what you try and configure, then I can
probably give a better answer. I tried and explain it here:
http://www.elasticsearch.com/docs/elasticsearch/setup/configuration/, what
specifically do you struggle with so I can explain it better in the docs....

-shay.banon

On Tue, Mar 9, 2010 at 5:40 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Hiya

Does anybody have a real life example of an elasticsearch config file.

I'm somewhat confused about what goes into the config file, and what
gets specified when (eg) creating an index.

thanks

Clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

3

Hiya

If you can give an example of what you try and configure, then I can
probably give a better answer. I tried and explain it here:
http://www.elasticsearch.com/docs/elasticsearch/setup/configuration/,
what specifically do you struggle with so I can explain it better in
the docs....

Yeah sorry, crap question :slight_smile:

OK, for instance, do the following settings make sense. I realise that
some are the same as the defaults (and thus redundant) but I wanted to
be sure that I was putting them in the right place.

cluster:
    name:               MyCluster

network:
    bindHost:           192.168.5.100
    publishHost:        192.168.5.100

node:
    data:               true

http:
    enabled:            true


gateway:
    type:               fs
    fs:             
        location:       /opt/elasticsearch/data/cluster

index:
    gateway:
        type:           fs
        fs:             
            location:   /opt/elasticsearch/data/index

    store:
        type:           memory
        cacheSize:      100m

Of course, when I create an index, I can set the gateway.fs.location to
a different directory, which would make sense, no?

For things like mappings, would/could you put them in the config file,
would you just put_mapping via the API?

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

4

Answered below:

On Tue, Mar 9, 2010 at 7:00 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Hiya

If you can give an example of what you try and configure, then I can
probably give a better answer. I tried and explain it here:
http://www.elasticsearch.com/docs/elasticsearch/setup/configuration/,
what specifically do you struggle with so I can explain it better in
the docs....

Yeah sorry, crap question :slight_smile:

OK, for instance, do the following settings make sense. I realise that
some are the same as the defaults (and thus redundant) but I wanted to
be sure that I was putting them in the right place.

cluster:
name: MyCluster

network:
bindHost: 192.168.5.100
publishHost: 192.168.5.100

node:
data: true

http:
enabled: true

gateway:
type: fs
fs:
location: /opt/elasticsearch/data/cluster

index:
gateway:
type: fs
fs:
location: /opt/elasticsearch/data/index

   store:
       type:           memory
       cacheSize:      100m

The store one is wrong, and I would play with the bufferSize as well:

store:
type: memory
memory:
cacheSize: 100m
bufferSize: 10k

Of course, when I create an index, I can set the gateway.fs.location to
a different directory, which would make sense, no?

Depends. When you explicitly set the one for the index, then you need to
make sure that you don't have two indices pointing to the same location. If
you don't specify one, then the global gateway location will be used, with
the index name appended as a sub directory. So I would say just leave the
global one to be fs, and let the index level ones "inherit it" (the index
will automatically use fs gateway if the global gateway is fs).

For things like mappings, would/could you put them in the config file,
would you just put_mapping via the API?

Currently, you can only place do it through the API. There is no option for
the cluster to start up and read mappings from a specific location. Of
course, there is also the question of which mappings in such a case apply to
which index...

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

5

Depends. When you explicitly set the one for the index, then you need
to make sure that you don't have two indices pointing to the same
location. If you don't specify one, then the global gateway location
will be used, with the index name appended as a sub directory. So I
would say just leave the global one to be fs, and let the index level
ones "inherit it" (the index will automatically use fs gateway if the
global gateway is fs).

OK - makes sense.

So if I wanted to change the mapping for an existing index/type, I could
do something like this:

  • create index 'temp_index'
  • put mapping 'new_type'
  • put other mappings from 'index' (except for 'old_type')
  • foreach document in 'index'
    • save to temp_index
  • stop nodes
  • mv index old_index
  • mv temp_index index
  • restart nodes

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

6
  • mv index old_index
  • mv temp_index index

I assume you mean mv the gateway location? Yes, this should work, though I
don't like the fact that you need to stop the nodes and start them again!. I
guess once elasticsearch will have the ability to "rename", or at the very
least "alias/ln -s" indices then you won't need to restart nodes as you can
create work with the aliases names and then "re-alias".

Also, iterating over the documents and reindexing should be simpler with
elasticsearch. One step towards this is the ability to continue to scroll
search requests (which I am working on now). The second is to have an API
for that.

-shay.banon

On Tue, Mar 9, 2010 at 7:37 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Depends. When you explicitly set the one for the index, then you need
to make sure that you don't have two indices pointing to the same
location. If you don't specify one, then the global gateway location
will be used, with the index name appended as a sub directory. So I
would say just leave the global one to be fs, and let the index level
ones "inherit it" (the index will automatically use fs gateway if the
global gateway is fs).

OK - makes sense.

So if I wanted to change the mapping for an existing index/type, I could
do something like this:

  • create index 'temp_index'
  • put mapping 'new_type'
  • put other mappings from 'index' (except for 'old_type')
  • foreach document in 'index'
    • save to temp_index
  • stop nodes
  • mv index old_index
  • mv temp_index index
  • restart nodes

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

7

One last point, I am also slowly making the put_mapping API more smarter
with merging new definitions with old ones. Changing the type from integer
to string will probably not be supported (though you can use the new
multi_field mapping to handle that), but changing things like term vectors
and such should be supported. Of course, adding new mappings is supported
already.

-shay.banon

On Tue, Mar 9, 2010 at 7:52 PM, Shay Banon shay.banon@elasticsearch.comwrote:

  • mv index old_index
  • mv temp_index index

I assume you mean mv the gateway location? Yes, this should work, though I
don't like the fact that you need to stop the nodes and start them again!. I
guess once elasticsearch will have the ability to "rename", or at the very
least "alias/ln -s" indices then you won't need to restart nodes as you can
create work with the aliases names and then "re-alias".

Also, iterating over the documents and reindexing should be simpler with
elasticsearch. One step towards this is the ability to continue to scroll
search requests (which I am working on now). The second is to have an API
for that.

-shay.banon

On Tue, Mar 9, 2010 at 7:37 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Depends. When you explicitly set the one for the index, then you need
to make sure that you don't have two indices pointing to the same
location. If you don't specify one, then the global gateway location
will be used, with the index name appended as a sub directory. So I
would say just leave the global one to be fs, and let the index level
ones "inherit it" (the index will automatically use fs gateway if the
global gateway is fs).

OK - makes sense.

So if I wanted to change the mapping for an existing index/type, I could
do something like this:

  • create index 'temp_index'
  • put mapping 'new_type'
  • put other mappings from 'index' (except for 'old_type')
  • foreach document in 'index'
    • save to temp_index
  • stop nodes
  • mv index old_index
  • mv temp_index index
  • restart nodes

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

8

On Tue, 2010-03-09 at 19:54 +0200, Shay Banon wrote:

One last point, I am also slowly making the put_mapping API more
smarter with merging new definitions with old ones. Changing the type
from integer to string will probably not be supported (though you can
use the new multi_field mapping to handle that), but changing things
like term vectors and such should be supported. Of course, adding new
mappings is supported already.

w00t!

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.