Re: How to put mappings and still get dynamic mappings

Elastic Stack Elasticsearch
1

Setting the mapping in advance for those fields, and indexing will retain the dynamic nature for the rest of the document. Can you maybe gist your put mapping code, and how you detect the other fields are not being indexed?
On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how to set a few mappings manually on my schema and retain the automatic mappings.

The exact problem is that I've been using elastic for the past week or so completely with the dynamic schema mappings and everything was fine. But I now want to sometimes return a field with my query... So there are a few fields in the schema that I want to set stored to true on. For the rest of the fields, I want them to continue to get their mappings dynamically, since there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already indexed, obviously there is a merge conflict because it has already had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it doesn't seem to have any effect and checking the _mapping shows that stored still isn't set.
I've tried blowing away the index completely, then setting a mapping with the two fields I care about being stored in it, and then re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

2

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will retain the dynamic nature for the rest of the document. Can you maybe gist your put mapping code, and how you detect the other fields are not being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how to set a few mappings manually on my schema and retain the automatic mappings.

The exact problem is that I've been using elastic for the past week or so completely with the dynamic schema mappings and everything was fine. But I now want to sometimes return a field with my query... So there are a few fields in the schema that I want to set stored to true on. For the rest of the fields, I want them to continue to get their mappings dynamically, since there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already indexed, obviously there is a merge conflict because it has already had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it doesn't seem to have any effect and checking the _mapping shows that stored still isn't set.
I've tried blowing away the index completely, then setting a mapping with the two fields I care about being stored in it, and then re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

3

Hmm... I've definitely screwed something up then.

So I set the code mapping like:

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{
"block": {
"properties": {
"system": {
"store": "yes",
"type": "string"
},
"user": {
"store": "yes",
"type": "string"
}

    }
}

}'

Start elastic and my server. And then reindex the users (starting with all
the data deleted). I should say here just to confirm that when I index
without having done this... everything works perfectly. I just can't tell it
to store the fields I want it to. So I think that I'm indexing the data ok.
So, after indexing, I run the mapping call and get this:

$ curl -XGET http://localhost:9200/_mapping | pp
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
128 128 128 128 0 0 22483 0 --:--:-- --:--:-- --:--:--
32000
{
"blocks": {
"block": {
"properties": {
"system": {
"store": "yes",
"type": "string"
},
"user": {
"store": "yes",
"type": "string"
}
}
}
},
"users": {}
}

Normally, after indexing these will be filled out with a bunch of
properties. When I run a search, I get exceptions on the client side and the
elastic logs have a number of exceptions along the lines of:

Parse Failure [No mapping found for [created_dt]]

Thanks for your help,

Pete

On Mon, Dec 13, 2010 at 6:31 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Setting the mapping in advance for those fields, and indexing will retain
the dynamic nature for the rest of the document. Can you maybe gist your put
mapping code, and how you detect the other fields are not being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how to
set a few mappings manually on my schema and retain the automatic mappings.

The exact problem is that I've been using elastic for the past week or so
completely with the dynamic schema mappings and everything was fine. But I
now want to sometimes return a field with my query... So there are a few
fields in the schema that I want to set stored to true on. For the rest of
the fields, I want them to continue to get their mappings dynamically, since
there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already indexed,
obviously there is a merge conflict because it has already had indexed those
fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it doesn't
seem to have any effect and checking the _mapping shows that stored still
isn't set.
I've tried blowing away the index completely, then setting a mapping with
the two fields I care about being stored in it, and then re-indexing, but
then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

4

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how
to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past week or
so completely with the dynamic schema mappings and everything was fine. But
I now want to sometimes return a field with my query... So there are a few
fields in the schema that I want to set stored to true on. For the rest of
the fields, I want them to continue to get their mappings dynamically, since
there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it doesn't
seem to have any effect and checking the _mapping shows that stored still
isn't set.
I've tried blowing away the index completely, then setting a mapping
with the two fields I care about being stored in it, and then re-indexing,
but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

5

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will retain
the dynamic nature for the rest of the document. Can you maybe gist your put
mapping code, and how you detect the other fields are not being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how to
set a few mappings manually on my schema and retain the automatic mappings.

The exact problem is that I've been using elastic for the past week or
so completely with the dynamic schema mappings and everything was fine. But
I now want to sometimes return a field with my query... So there are a few
fields in the schema that I want to set stored to true on. For the rest of
the fields, I want them to continue to get their mappings dynamically, since
there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it doesn't
seem to have any effect and checking the _mapping shows that stored still
isn't set.
I've tried blowing away the index completely, then setting a mapping
with the two fields I care about being stored in it, and then re-indexing,
but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

6

Not sure exactly what you want. This gist has the mapping that is generated
dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how
to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past week or
so completely with the dynamic schema mappings and everything was fine. But
I now want to sometimes return a field with my query... So there are a few
fields in the schema that I want to set stored to true on. For the rest of
the fields, I want them to continue to get their mappings dynamically, since
there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it doesn't
seem to have any effect and checking the _mapping shows that stored still
isn't set.
I've tried blowing away the index completely, then setting a mapping
with the two fields I care about being stored in it, and then re-indexing,
but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

7

I don't see where you put the mapping. You can explicitly set mappings for
specific fields, and those mappings will apply for them, the rest will be
dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is generated
dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how
to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past week or
so completely with the dynamic schema mappings and everything was fine. But
I now want to sometimes return a field with my query... So there are a few
fields in the schema that I want to set stored to true on. For the rest of
the fields, I want them to continue to get their mappings dynamically, since
there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a mapping
with the two fields I care about being stored in it, and then re-indexing,
but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

8

Oh, I see. Sorry for the confusion. I added the mappings for the fields that
I cared about with the curl command from the previous email (pasted below).
Does it make a difference whether I do it in java versus with the REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{
"block": {
"properties": {
"system": {
"store": "yes",
"type": "string"
},
"user": {
"store": "yes",
"type": "string"
}

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set mappings for
specific fields, and those mappings will apply for them, the rest will be
dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is generated

dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure how
to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past week
or so completely with the dynamic schema mappings and everything was fine.
But I now want to sometimes return a field with my query... So there are a
few fields in the schema that I want to set stored to true on. For the rest
of the fields, I want them to continue to get their mappings dynamically,
since there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a mapping
with the two fields I care about being stored in it, and then re-indexing,
but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

9

It doesn't make a difference, but note that those mappings will only be
applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the fields
that I cared about with the curl command from the previous email (pasted
below). Does it make a difference whether I do it in java versus with the
REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set mappings for
specific fields, and those mappings will apply for them, the rest will be
dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure
how to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past week
or so completely with the dynamic schema mappings and everything was fine.
But I now want to sometimes return a field with my query... So there are a
few fields in the schema that I want to set stored to true on. For the rest
of the fields, I want them to continue to get their mappings dynamically,
since there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a mapping
with the two fields I care about being stored in it, and then re-indexing,
but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

10

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added. I
just updated to 0.13.1 to see if that would help. Sorry, if I'm being obtuse
here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only be
applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the fields

that I cared about with the curl command from the previous email (pasted
below). Does it make a difference whether I do it in java versus with the
REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set mappings
for specific fields, and those mappings will apply for them, the rest will
be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've been
using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure
how to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past week
or so completely with the dynamic schema mappings and everything was fine.
But I now want to sometimes return a field with my query... So there are a
few fields in the schema that I want to set stored to true on. For the rest
of the fields, I want them to continue to get their mappings dynamically,
since there are a lot of them and more may be added in the future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

11

Sounds good. And if you get the mapping back do you see the stored mapping
def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added. I
just updated to 0.13.1 to see if that would help. Sorry, if I'm being obtuse
here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only be
applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the fields

that I cared about with the curl command from the previous email (pasted
below). Does it make a difference whether I do it in java versus with the
REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set mappings
for specific fields, and those mappings will apply for them, the rest will
be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've
been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure
how to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past
week or so completely with the dynamic schema mappings and everything was
fine. But I now want to sometimes return a field with my query... So there
are a few fields in the schema that I want to set stored to true on. For the
rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

12

Hey,

I got it working... it looks like the mistake I was making was naming my
mapping the same thing as the type. I don't know if that matters or not, but
that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored mapping
def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added. I
just updated to 0.13.1 to see if that would help. Sorry, if I'm being obtuse
here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only be
applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the fields

that I cared about with the curl command from the previous email (pasted
below). Does it make a difference whether I do it in java versus with the
REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set mappings
for specific fields, and those mappings will apply for them, the rest will
be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've
been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and indexing will
retain the dynamic nature for the rest of the document. Can you maybe gist
your put mapping code, and how you detect the other fields are not being
indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't figure
how to set a few mappings manually on my schema and retain the automatic
mappings.

The exact problem is that I've been using elastic for the past
week or so completely with the dynamic schema mappings and everything was
fine. But I now want to sometimes return a field with my query... So there
are a few fields in the schema that I want to set stored to true on. For the
rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data already
indexed, obviously there is a merge conflict because it has already had
indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

13

Great that its working. What do you mean that you named the mapping same
thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming my
mapping the same thing as the type. I don't know if that matters or not, but
that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored mapping
def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added. I
just updated to 0.13.1 to see if that would help. Sorry, if I'm being obtuse
here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only be
applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the fields

that I cared about with the curl command from the previous email (pasted
below). Does it make a difference whether I do it in java versus with the
REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set mappings
for specific fields, and those mappings will apply for them, the rest will
be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've
been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and indexing
will retain the dynamic nature for the rest of the document. Can you maybe
gist your put mapping code, and how you detect the other fields are not
being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the past
week or so completely with the dynamic schema mappings and everything was
fine. But I now want to sometimes return a field with my query... So there
are a few fields in the schema that I want to set stored to true on. For the
rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

14

So I was doing something like

curl -XPUT $ES_HOST/blocks/block/_mapping -d '
{
"block" : { ....

The only change that I think I made was to change the "block" in the json
body to something like "initial-block-mapping".

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Great that its working. What do you mean that you named the mapping same
thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming my
mapping the same thing as the type. I don't know if that matters or not, but
that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored
mapping def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added. I
just updated to 0.13.1 to see if that would help. Sorry, if I'm being obtuse
here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only be
applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the fields

that I cared about with the curl command from the previous email (pasted
below). Does it make a difference whether I do it in java versus with the
REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set
mappings for specific fields, and those mappings will apply for them, the
rest will be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've
been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and indexing
will retain the dynamic nature for the rest of the document. Can you maybe
gist your put mapping code, and how you detect the other fields are not
being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the past
week or so completely with the dynamic schema mappings and everything was
fine. But I now want to sometimes return a field with my query... So there
are a few fields in the schema that I want to set stored to true on. For the
rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data, it
doesn't seem to have any effect and checking the _mapping shows that stored
still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

15

Actually, I take that back. I just had the problem recur. I had switched the
number of replicas to be zero since I'm testing on only 1 node anyway. When
I switched it to 2 replicas, blew away the data, and re-indexed, I ran into
the same problem. I just switched back to 0 replicas and it's fine again.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

So I was doing something like

curl -XPUT $ES_HOST/blocks/block/_mapping -d '
{
"block" : { ....

The only change that I think I made was to change the "block" in the json
body to something like "initial-block-mapping".

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Great that its working. What do you mean that you named the mapping same
thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming my
mapping the same thing as the type. I don't know if that matters or not, but
that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored
mapping def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added.
I just updated to 0.13.1 to see if that would help. Sorry, if I'm being
obtuse here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only
be applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the

fields that I cared about with the curl command from the previous email
(pasted below). Does it make a difference whether I do it in java versus
with the REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set
mappings for specific fields, and those mappings will apply for them, the
rest will be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because I've
been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and indexing
will retain the dynamic nature for the rest of the document. Can you maybe
gist your put mapping code, and how you detect the other fields are not
being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the past
week or so completely with the dynamic schema mappings and everything was
fine. But I now want to sometimes return a field with my query... So there
are a few fields in the schema that I want to set stored to true on. For the
rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data,
it doesn't seem to have any effect and checking the _mapping shows that
stored still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

16

The number of replicas should not be relevant to what you do. Can you "curl"
recreate this?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Actually, I take that back. I just had the problem recur. I had switched
the number of replicas to be zero since I'm testing on only 1 node anyway.
When I switched it to 2 replicas, blew away the data, and re-indexed, I ran
into the same problem. I just switched back to 0 replicas and it's fine
again.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

So I was doing something like

curl -XPUT $ES_HOST/blocks/block/_mapping -d '
{
"block" : { ....

The only change that I think I made was to change the "block" in the json
body to something like "initial-block-mapping".

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Great that its working. What do you mean that you named the mapping same
thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming my
mapping the same thing as the type. I don't know if that matters or not, but
that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored
mapping def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't added.
I just updated to 0.13.1 to see if that would help. Sorry, if I'm being
obtuse here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only
be applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the

fields that I cared about with the curl command from the previous email
(pasted below). Does it make a difference whether I do it in java versus
with the REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set
mappings for specific fields, and those mappings will apply for them, the
rest will be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because
I've been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and indexing
will retain the dynamic nature for the rest of the document. Can you maybe
gist your put mapping code, and how you detect the other fields are not
being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the
past week or so completely with the dynamic schema mappings and everything
was fine. But I now want to sometimes return a field with my query... So
there are a few fields in the schema that I want to set stored to true on.
For the rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data,
it doesn't seem to have any effect and checking the _mapping shows that
stored still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

17

Sure. I was able to re-create a problem with commands in this
gisthttps://gist.github.com/744065
.

2010/12/16 Shay Banon shay.banon@elasticsearch.com

The number of replicas should not be relevant to what you do. Can you
"curl" recreate this?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Actually, I take that back. I just had the problem recur. I had switched
the number of replicas to be zero since I'm testing on only 1 node anyway.
When I switched it to 2 replicas, blew away the data, and re-indexed, I ran
into the same problem. I just switched back to 0 replicas and it's fine
again.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

So I was doing something like

curl -XPUT $ES_HOST/blocks/block/_mapping -d '
{
"block" : { ....

The only change that I think I made was to change the "block" in the json
body to something like "initial-block-mapping".

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Great that its working. What do you mean that you named the mapping same
thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming
my mapping the same thing as the type. I don't know if that matters or not,
but that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored
mapping def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't
added. I just updated to 0.13.1 to see if that would help. Sorry, if I'm
being obtuse here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will only
be applied if there was no previous docs before indexed with that field. So,
delete the index, create a new one with the mappings, and then start
indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the

fields that I cared about with the curl command from the previous email
(pasted below). Does it make a difference whether I do it in java versus
with the REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set
mappings for specific fields, and those mappings will apply for them, the
rest will be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because
I've been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and indexing
will retain the dynamic nature for the rest of the document. Can you maybe
gist your put mapping code, and how you detect the other fields are not
being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the
past week or so completely with the dynamic schema mappings and everything
was fine. But I now want to sometimes return a field with my query... So
there are a few fields in the schema that I want to set stored to true on.
For the rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my data,
it doesn't seem to have any effect and checking the _mapping shows that
stored still isn't set.
I've tried blowing away the index completely, then setting a
mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

18

I don't really understand how this recreates the mapping problem? The reason
you can't index is that you created 2 replicas in the index, and started one
nodes, and, by default, a write (index) only works when there is a quorum of
shards active within the shard replication group. In your case, there is 1
shard active per replication groups, and 2 replicas not allocated (because
replicas are not allocated on the same node), and the quorum is 2.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Sure. I was able to re-create a problem with commands in this gisthttps://gist.github.com/744065
.

2010/12/16 Shay Banon shay.banon@elasticsearch.com

The number of replicas should not be relevant to what you do. Can you

"curl" recreate this?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Actually, I take that back. I just had the problem recur. I had switched
the number of replicas to be zero since I'm testing on only 1 node anyway.
When I switched it to 2 replicas, blew away the data, and re-indexed, I ran
into the same problem. I just switched back to 0 replicas and it's fine
again.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

So I was doing something like

curl -XPUT $ES_HOST/blocks/block/_mapping -d '
{
"block" : { ....

The only change that I think I made was to change the "block" in the
json body to something like "initial-block-mapping".

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Great that its working. What do you mean that you named the mapping
same thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming
my mapping the same thing as the type. I don't know if that matters or not,
but that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored
mapping def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't
added. I just updated to 0.13.1 to see if that would help. Sorry, if I'm
being obtuse here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will
only be applied if there was no previous docs before indexed with that
field. So, delete the index, create a new one with the mappings, and then
start indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the

fields that I cared about with the curl command from the previous email
(pasted below). Does it make a difference whether I do it in java versus
with the REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set
mappings for specific fields, and those mappings will apply for them, the
rest will be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that is

generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd wrote:

Thanks... that's a great pointer. I'm not seeing that because
I've been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and
indexing will retain the dynamic nature for the rest of the document. Can
you maybe gist your put mapping code, and how you detect the other fields
are not being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd
wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the
past week or so completely with the dynamic schema mappings and everything
was fine. But I now want to sometimes return a field with my query... So
there are a few fields in the schema that I want to set stored to true on.
For the rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my
data, it doesn't seem to have any effect and checking the _mapping shows
that stored still isn't set.
I've tried blowing away the index completely, then setting
a mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete

19

Yeah, I didn't realize that, although it makes sense. I'm assuming that an
exception would have also been thrown in java then. It must have gotten
swallowed somewhere. In any case, thanks for your help.

Pete

2010/12/16 Shay Banon shay.banon@elasticsearch.com

I don't really understand how this recreates the mapping problem? The
reason you can't index is that you created 2 replicas in the index, and
started one nodes, and, by default, a write (index) only works when there is
a quorum of shards active within the shard replication group. In your case,
there is 1 shard active per replication groups, and 2 replicas not allocated
(because replicas are not allocated on the same node), and the quorum is 2.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Sure. I was able to re-create a problem with commands in this gisthttps://gist.github.com/744065
.

2010/12/16 Shay Banon shay.banon@elasticsearch.com

The number of replicas should not be relevant to what you do. Can you

"curl" recreate this?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Actually, I take that back. I just had the problem recur. I had switched
the number of replicas to be zero since I'm testing on only 1 node anyway.
When I switched it to 2 replicas, blew away the data, and re-indexed, I ran
into the same problem. I just switched back to 0 replicas and it's fine
again.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

So I was doing something like

curl -XPUT $ES_HOST/blocks/block/_mapping -d '
{
"block" : { ....

The only change that I think I made was to change the "block" in the
json body to something like "initial-block-mapping".

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Great that its working. What do you mean that you named the mapping
same thing as the type?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Hey,

I got it working... it looks like the mistake I was making was naming
my mapping the same thing as the type. I don't know if that matters or not,
but that appears to be the only difference.

Thanks for all the help!

Pete

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Sounds good. And if you get the mapping back do you see the stored
mapping def for those fields?

2010/12/16 Pete Aykroyd aykroyd@gmail.com

So that's what I did:

  1. I deleted data/*
  2. started elastic
  3. created the index with curl
  4. used that curl command to put the mappings
  5. and then indexed my docs

There were no errors while indexing, but the new fields weren't
added. I just updated to 0.13.1 to see if that would help. Sorry, if I'm
being obtuse here but it seems like I'm doing the right thing.

2010/12/15 Shay Banon shay.banon@elasticsearch.com

It doesn't make a difference, but note that those mappings will
only be applied if there was no previous docs before indexed with that
field. So, delete the index, create a new one with the mappings, and then
start indexing.

2010/12/16 Pete Aykroyd aykroyd@gmail.com

Oh, I see. Sorry for the confusion. I added the mappings for the

fields that I cared about with the curl command from the previous email
(pasted below). Does it make a difference whether I do it in java versus
with the REST API?

Thanks,

Pete

curl -XPUT 'http://localhost:9200/blocks/block/_mapping' -d '
{

"block": {
    "properties": {
        "system": {
            "store": "yes",
            "type": "string"
        },
        "user": {
            "store": "yes",
            "type": "string"
        }

    }
}

}'

2010/12/15 Shay Banon shay.banon@elasticsearch.com

I don't see where you put the mapping. You can explicitly set
mappings for specific fields, and those mappings will apply for them, the
rest will be dynamic.

2010/12/15 Pete Aykroyd aykroyd@gmail.com

Not sure exactly what you want. This gist has the mapping that

is generated dynamically and the code that does the indexing.

https://gist.github.com/742513

2010/12/15 Shay Banon shay.banon@elasticsearch.com

Can you gist a full recreation?

On Wednesday, December 15, 2010 at 1:12 AM, Pete Aykroyd
wrote:

Thanks... that's a great pointer. I'm not seeing that because
I've been using 0.13.0. Still curious about this mapping issue though.

Pete

On Tue, Dec 14, 2010 at 1:39 AM, Medcl medcl@163.com wrote:

as i know,es now can direct use _source's item as field's
value,check
it out,
Search Fields: If a field is not stored, automatically extract it from _source (without the need for _source prefix) · Issue #562 · elastic/elasticsearch · GitHub

On 12月14日, 上午10时31分, Shay Banon shay.ba...@elasticsearch.com
wrote:

Setting the mapping in advance for those fields, and
indexing will retain the dynamic nature for the rest of the document. Can
you maybe gist your put mapping code, and how you detect the other fields
are not being indexed?

On Tuesday, December 14, 2010 at 12:40 AM, Pete Aykroyd
wrote:

Hi,

I think I'm probably missing something obvious but I can't
figure how to set a few mappings manually on my schema and retain the
automatic mappings.

The exact problem is that I've been using elastic for the
past week or so completely with the dynamic schema mappings and everything
was fine. But I now want to sometimes return a field with my query... So
there are a few fields in the schema that I want to set stored to true on.
For the rest of the fields, I want them to continue to get their mappings
dynamically, since there are a lot of them and more may be added in the
future.

If try to put a mapping for just that the field with data
already indexed, obviously there is a merge conflict because it has already
had indexed those fields with stored = false.
If I set ignore_conflicts to true and then re-index my
data, it doesn't seem to have any effect and checking the _mapping shows
that stored still isn't set.
I've tried blowing away the index completely, then setting
a mapping with the two fields I care about being stored in it, and then
re-indexing, but then nothing else is mapped.

What's the right way to do this?

Thanks,

Pete