Default-mapping.json

Elastic Stack Elasticsearch
1

Hi,
I've create default-mapping.json file with this content:
{
"default" : {
"_source" : {"enabled" : false}
}
}

in the elasticsearch / config folder but it seems that _source field is
stored anyway.
Probably I'm missing something or I did something wrong but don't have any
idea why default mapping didn't work in my case.

Any suggestions are more than welcome.

Thank You and Best Regards.

2

It should work, just placed defualt-mapping.json within the config folder,
and ran:

curl -XPUT localhost:9200/test/type/1 -d '{"test" : "value"}'
curl localhost:9200/test/_mapping?pretty=1

And I can see that the mapping does not have source enabled.

Do you have special paths configured? Maybe config is pointing to another
location?

On Fri, Dec 9, 2011 at 9:36 PM, slavag slavago@gmail.com wrote:

Hi,
I've create default-mapping.json file with this content:
{
"default" : {
"_source" : {"enabled" : false}
}
}

in the elasticsearch / config folder but it seems that _source field is
stored anyway.
Probably I'm missing something or I did something wrong but don't have any
idea why default mapping didn't work in my case.

Any suggestions are more than welcome.

Thank You and Best Regards.

3

Hi Shay,

Thank You for quick response,
Sorry, I'm used to work with java, but now tried curl with command you sent
(first one) and this error that I got :
{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected character (''' (code 39)): expected a valid
value (number, String, array, object, 'true', 'false' or 'null')\n at
[Source: [B@1636731; line: 1, column: 2]]; ","status":400}

As for any special path configured - the answer is no, i put only
default-mapping.json and that's all.

Best Regards.

4

When did you get this failure? when you indexed the document? There isn't
really a difference between the Java index API and the curl (http REST)
API. How do you know that your default-mapping.json does not work? The curl
sample I gave helps to show that it works.

On Sat, Dec 10, 2011 at 12:43 AM, slavag slavago@gmail.com wrote:

Hi Shay,

Thank You for quick response,
Sorry, I'm used to work with java, but now tried curl with command you
sent (first one) and this error that I got :
{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected character (''' (code 39)): expected a valid
value (number, String, array, object, 'true', 'false' or 'null')\n at
[Source: [B@1636731; line: 1, column: 2]]; ","status":400}

As for any special path configured - the answer is no, i put only
default-mapping.json and that's all.

Best Regards.

5

This failure is happened on this command curl -XPUT
localhost:9200/test/type/1 -d '{"test" : "value"}'
But, this is not the issue, I know that default-mapping.json doesn't work ,
when I created (JAVA code) the index and on search i got the _source field.
Until I created a new index (manually and with mapping where I made _source
disabled) and then search doesn't returned the _source.

Thank You.

6

It seems to work for me, so if you can provide a details step by step (from
a fresh installation) process of what you do, so I can follow it here, I
can try and help.

On Sat, Dec 10, 2011 at 9:49 AM, slavag slavago@gmail.com wrote:

This failure is happened on this command curl -XPUT
localhost:9200/test/type/1 -d '{"test" : "value"}'
But, this is not the issue, I know that default-mapping.json doesn't work
, when I created (JAVA code) the index and on search i got the _source
field.
Until I created a new index (manually and with mapping where I made
_source disabled) and then search doesn't returned the _source.

Thank You.

7

Yes, sure.
Fresh 0.18.5 download and extract.
Make a file default-mapping.json with :
{
"default" : {
"_source" : {"enabled" : false}
}
}

put it under config folder - no any other changes.
executed elastic search (btw Win 7 - 64).

Created index automatically by indexing data (using Java) and tried to
search (Java as well) - got _source field.
Stopped ES
Deleted ES data.
Started ES.
Created Index manually (using admin().indices().create(new
CreateIndexRequest("testindex").mapping(type, mapping)).

The mapping is next :
XContentBuilder data = jsonBuilder()
.startObject()
.startObject("type")
.startObject("_source")
.field("enabled", false)
.endObject()
.endObject()
.endObject();

Tried search - no _source field.

Best Regards.

8

May be I need to put default-mapping.json to config/mappings/_default ?

9

No, config is the right place for it. I just double checked with 0.18.5,
and it works well. Can you try the steps I mentioned with curl and double
check that what I do works for you (I want to take your Java code out of
the picture)?

  1. Place default-mapping.json under config/
  2. Run curl -XPUT localhost:9200/test/type/1 -d '{"test" : "value"}'
  3. Run curl localhost:9200/test/_mapping?pretty=1

After that, you should see in the mapping that source is disabled. If it
works for you, can you gist the Java code you use?

On Mon, Dec 12, 2011 at 7:35 AM, slavag slavago@gmail.com wrote:

May be I need to put default-mapping.json to config/mappings/_default ?

10

Hi,
Error when running curl -XPUT localhost:9200/test/type/1 -d '{"test" :
"value"}'
Could be that something wrong with curl ??

I see this error from the curl output :
{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected character (''' (code 39)): expected a valid
value (number, String, array, object, 'true', 'false' or 'null')\n
at [Source: [B@bcfa38; line: 1, column: 2]]; ","status":400}curl: (7)
couldn't connect to host
curl: (3) [globbing] unmatched close brace/bracket at pos 6

The exception on ES server side is :
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:480)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:414)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:302)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:181)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.common.jackson.JsonParseException: Unexpected
character (''' (code 39)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: [B@bcfa38; line: 1, column: 2]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:2084)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._nextTokenNotInObject(Utf8StreamParser.java:600)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:486)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:46)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:434)
... 8 more

Best Regards.

11

Where are you executing the curl request from? Are you using windows
(cygwin)?

On Mon, Dec 12, 2011 at 6:13 PM, slavag slavago@gmail.com wrote:

Hi,
Error when running curl -XPUT localhost:9200/test/type/1 -d '{"test" :
"value"}'
Could be that something wrong with curl ??

I see this error from the curl output :
{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected character (''' (code 39)): expected a valid
value (number, String, array, object, 'true', 'false' or 'null')\n
at [Source: [B@bcfa38; line: 1, column: 2]]; ","status":400}curl: (7)
couldn't connect to host
curl: (3) [globbing] unmatched close brace/bracket at pos 6

The exception on ES server side is :
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:480)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:414)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:302)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:181)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:487)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.common.jackson.JsonParseException: Unexpected
character (''' (code 39)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: [B@bcfa38; line: 1, column: 2]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:2084)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._nextTokenNotInObject(Utf8StreamParser.java:600)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:486)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:46)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:434)
... 8 more

Best Regards.

12

It curl compiled for windows (not cygwin) . And executing from windows
console.

13

I have very bad issues with curl for windows.
Be careful with " and '

Today, curl for windows drives me crazy because _analyze was not found, but I get no problem with curl under Linux.

Sometimes, it's better with curl for Windows to use the -d @filename option to push your datas

HTH
David :wink:
@dadoonet

Le 12 déc. 2011 à 21:22, slavag slavago@gmail.com a écrit :

It curl compiled for windows (not cygwin) . And executing from windows console.

14

On 12 December 2011 21:31, David Pilato david@pilato.fr wrote:

I have very bad issues with curl for windows.
Be careful with " and '

Today, curl for windows drives me crazy because _analyze was not found, but I get no problem with curl under Linux.

Sometimes, it's better with curl for Windows to use the -d @filename option to push your datas

HTH
David :wink:
@dadoonet

win32 quoting is somewhat arcane and inconsistent.

the canonical reference is at
The Windows NT Command Shell | Microsoft Learn and I put a
few practical notes on the couchdb wiki at
http://wiki.apache.org/couchdb/Quirks_on_Windows

A+
Dave

15

WOW !!!!!!!!!!!!!
With file it works.
Shay, so with input file I executed curl -XPUT localhost:9200/test/type/1
-d '{"test" : "value"}'
response was
: {"ok":true,"_index":"test","_type":"type","_id":"1","_version":1}
And response on curl localhost:9200/test/_mapping?pretty=1 :

{
"test" : {
"type" : {
"properties" : {
"test" : {
"type" : "string"
}
}
}
}
}

So , I don't see _source here and when I'm executing curl
localhost:9200/myindex/_mapping?pretty=1 I'm also not see
_source field, but when I'm making search in java code :
SearchRequestBuilder searchRequestBuilder = client.prepareSearch("myindex");
searchRequestBuilder.setSearchType(SearchType.DFS_QUERY_THEN_FETCH);
searchRequestBuilder.setQuery(textQuery( "_all", "slava");
searchRequestBuilder.setFrom(0).setSize(60).setExplain(false);
SearchResponse response = searchRequestBuilder.execute().actionGet();

The result is :
"hits" : {
"total" : 892,
"max_score" : 1.1002213,
"hits" : [ {
"_index" : "myindex",
"_type" : "abc",
"_id" : "47011109-c4e9-4072-8b0c-2ba78c2063d0",
"_score" : 1.1002213, "_source" :
{"date":"20111212203822","file":"Users-@-@-0}
}

And _source field is here.
So , may be I'm missing something ?

Best Regards.

16

Mmmm, the only other reason that I can think it might happen is that the
node you start joins another cluster (using multicast discovery), that does
not have this configuration. Can you check if it happens (the log will
output if it joins another cluster). You can disable multicast by
setting: discovery.zen.ping.multicast.enabled: false in the config file.

On Mon, Dec 12, 2011 at 11:12 PM, slavag slavago@gmail.com wrote:

WOW !!!!!!!!!!!!!
With file it works.
Shay, so with input file I executed curl -XPUT localhost:9200/test/type/1
-d '{"test" : "value"}'
response was
: {"ok":true,"_index":"test","_type":"type","_id":"1","_version":1}
And response on curl localhost:9200/test/_mapping?pretty=1 :

{
"test" : {
"type" : {
"properties" : {
"test" : {
"type" : "string"
}
}
}
}
}

So , I don't see _source here and when I'm executing curl
localhost:9200/myindex/_mapping?pretty=1 I'm also not see
_source field, but when I'm making search in java code :
SearchRequestBuilder searchRequestBuilder =
client.prepareSearch("myindex");
searchRequestBuilder.setSearchType(SearchType.DFS_QUERY_THEN_FETCH);
searchRequestBuilder.setQuery(textQuery( "_all", "slava");
searchRequestBuilder.setFrom(0).setSize(60).setExplain(false);
SearchResponse response = searchRequestBuilder.execute().actionGet();

The result is :
"hits" : {
"total" : 892,
"max_score" : 1.1002213,
"hits" : [ {
"_index" : "myindex",
"_type" : "abc",
"_id" : "47011109-c4e9-4072-8b0c-2ba78c2063d0",
"_score" : 1.1002213, "_source" :
{"date":"20111212203822","file":"Users-@-@-0}
}

And _source field is here.
So , may be I'm missing something ?

Best Regards.

17

Actually the computer is no part of any network - home computer. So I'm
doubt is it part of any cluster at all.
Btw, when I'm explicitly create an index with mapping (java) code then
_source field is not returned by search.

Any suggestions ?

18

Shay , any suggestion about my problem ?

Thank You.

19

I am not really sure..., I don't see where the problem is. Can someone else
try and recreate this problem, or verify that its not happening? I will
repeat the recreation:

  1. Use a fresh elasticsearch 0.18.5 installation.
  2. Place a file called default-mapping.json under config with the following
    content:

{
"default" : {
"_source" : {"enabled" : false}
}
}

  1. Index a sample document: curl -XPUT localhost:9200/test/type/1 -d
    '{"test" : "value"}'
  2. Execute: curl localhost:9200/test/_mapping?pretty=1, and check if it has
    source enabled or not in the mapping returned.

Thanks!

On Tue, Dec 13, 2011 at 11:58 PM, slavag slavago@gmail.com wrote:

Shay , any suggestion about my problem ?

Thank You.

20

Shay,
I did again what you said (from 1-4) and indeed in 4. the _source is not
returned.
The response on 4. is :

{
"test" : {
"type" : {
"properties" : {
"test" : {
"type" : "string"
}
}
}
}
}

BUT, when I did : curl -XGET localhost:9200/test/_search?pretty=true -d '
{
"query" : {
"term" : { "test" : "value" }
}
}
'
The response was :

{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.30685282,
"hits" : [ {
"_index" : "test",
"_type" : "type",
"_id" : "1",
"_score" : 0.30685282, "_source" : {"test" : "value"}
} ]
}
}

So you can see the _source field is returned.
Please advise.

Best Regards.