Wrong dynamic template or issue 1181?

Elastic Stack Elasticsearch
1

Hi,

I'm having problems when upgrading to 0.18.* (tried all versions, did
not occur in <0.18) and it seems to me I was either bitten via:

or via my template:
{ "tweet" : {
"dynamic_templates" : [
{
"template_1" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : { "type" : "string", "omit_norms" :
"yes" }
}
}
],
(no mapping for my date field called 'dt')
}}

When I remove this template all is fine but when not and I'm sorting
by my date field 'dt' I do not get any documents**:

count:2
hits:2
hits sorted:0

But why is now a date object of type string? Or is this related to the
issue where Shay says: "Will revert this, as its tricky without
changing default behavior now that I think about it. Need to think of
a better way to suport optional timezone in this case." So is this
issue already reverted (as it is closed) or is it in the 0.18
releases?

Peter.

Settings s = ImmutableSettings.settingsBuilder().put("cluster.name",
CLUSTER).build();
TransportClient client = new TransportClient(s);
client.addTransportAddress(new InetSocketTransportAddress("127.0.0.1",
PORT));
try {
client.admin().indices().create(new
CreateIndexRequest("tmp")).actionGet();
} catch (IndexAlreadyExistsException ex) {
}
client.prepareDeleteByQuery("tmp").setTypes("tweet").setQuery(QueryBuilders.matchAllQuery());
client.admin().indices().refresh(new RefreshRequest("tmp"));

BulkRequest br = new BulkRequest();
XContentBuilder cb =
JsonXContent.contentBuilder().startObject();
cb.field("dt", new Date(new Date().getTime() - 10000));
cb.field("text", "java");
br.add(new IndexRequest("tmp").id("1").type("tweet").source(cb));

cb = JsonXContent.contentBuilder().startObject();
cb.field("dt", new Date());
cb.field("text", "java");
br.add(new IndexRequest("tmp").id("2").type("tweet").source(cb));
client.bulk(br).actionGet();

client.admin().indices().refresh(new RefreshRequest("tmp"));
System.out.println("count:" + client.prepareCount("tmp").
execute().actionGet().count());
System.out.println("hits:" + client.prepareSearch("tmp").
setQuery(QueryBuilders.queryString("java").field("text")).
execute().actionGet().hits().totalHits());

System.out.println("hits sorted:" + client.prepareSearch("tmp").
setQuery(QueryBuilders.queryString("java").field("text")).
addSort("dt", SortOrder.DESC).
execute().actionGet().hits().totalHits());

2

ok, when I'm explicitely map the date field then it returns:
count:0
hits:2
hits sorted:2

for the count query I need to set the type => then all is fine (all
count==2) ... although this is not expected behaviour, right :slight_smile: ?

Peter.

On 5 Nov., 14:08, Karussell tableyourt...@googlemail.com wrote:

Hi,

I'm having problems when upgrading to 0.18.* (tried all versions, did
not occur in <0.18) and it seems to me I was either bitten via:Mapping: Root object non ISO date formats to support timezone · Issue #1181 · elastic/elasticsearch · GitHub

or via my template:
{ "tweet" : {
"dynamic_templates" : [
{
"template_1" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : { "type" : "string", "omit_norms" :
"yes" }
}
}
],
(no mapping for my date field called 'dt')

}}

When I remove this template all is fine but when not and I'm sorting
by my date field 'dt' I do not get any documents**:

count:2
hits:2
hits sorted:0

But why is now a date object of type string? Or is this related to the
issue where Shay says: "Will revert this, as its tricky without
changing default behavior now that I think about it. Need to think of
a better way to suport optional timezone in this case." So is this
issue already reverted (as it is closed) or is it in the 0.18
releases?

Peter.

Settings s = ImmutableSettings.settingsBuilder().put("cluster.name",
CLUSTER).build();
TransportClient client = new TransportClient(s);
client.addTransportAddress(new InetSocketTransportAddress("127.0.0.1",
PORT));
try {
client.admin().indices().create(new
CreateIndexRequest("tmp")).actionGet();} catch (IndexAlreadyExistsException ex) {
}

client.prepareDeleteByQuery("tmp").setTypes("tweet").setQuery(QueryBuilders.matchAllQuery());
client.admin().indices().refresh(new RefreshRequest("tmp"));

BulkRequest br = new BulkRequest();
XContentBuilder cb =
JsonXContent.contentBuilder().startObject();
cb.field("dt", new Date(new Date().getTime() - 10000));
cb.field("text", "java");
br.add(new IndexRequest("tmp").id("1").type("tweet").source(cb));

cb = JsonXContent.contentBuilder().startObject();
cb.field("dt", new Date());
cb.field("text", "java");
br.add(new IndexRequest("tmp").id("2").type("tweet").source(cb));
client.bulk(br).actionGet();

client.admin().indices().refresh(new RefreshRequest("tmp"));
System.out.println("count:" + client.prepareCount("tmp").
execute().actionGet().count());
System.out.println("hits:" + client.prepareSearch("tmp").
setQuery(QueryBuilders.queryString("java").field("text")).
execute().actionGet().hits().totalHits());

System.out.println("hits sorted:" + client.prepareSearch("tmp").
setQuery(QueryBuilders.queryString("java").field("text")).
addSort("dt", SortOrder.DESC).
execute().actionGet().hits().totalHits());

3

Heya,

Yea, its not related to this issue, but there was another issue where
dynamic templates on attachment did not work, which probably caused this
behavior. Basically, for a new field introduced, and its of type string, it
will first go and try and find a template for it (under string type). So,
in your case, it will be checked and applied as string, even though it
matches a date pattern. I opened an issue:
Mapping: Improve applying guessed types on dynamic templates · Issue #1446 · elastic/elasticsearch · GitHub.

-shay.banon

On Sat, Nov 5, 2011 at 3:13 PM, Karussell tableyourtime@googlemail.comwrote:

ok, when I'm explicitely map the date field then it returns:
count:0
hits:2
hits sorted:2

for the count query I need to set the type => then all is fine (all
count==2) ... although this is not expected behaviour, right :slight_smile: ?

Peter.

On 5 Nov., 14:08, Karussell tableyourt...@googlemail.com wrote:

Hi,

I'm having problems when upgrading to 0.18.* (tried all versions, did
not occur in <0.18) and it seems to me I was either bitten via:
Mapping: Root object non ISO date formats to support timezone · Issue #1181 · elastic/elasticsearch · GitHub

or via my template:
{ "tweet" : {
"dynamic_templates" : [
{
"template_1" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : { "type" : "string", "omit_norms" :
"yes" }
}
}
],
(no mapping for my date field called 'dt')

}}

When I remove this template all is fine but when not and I'm sorting
by my date field 'dt' I do not get any documents**:

count:2
hits:2
hits sorted:0

But why is now a date object of type string? Or is this related to the
issue where Shay says: "Will revert this, as its tricky without
changing default behavior now that I think about it. Need to think of
a better way to suport optional timezone in this case." So is this
issue already reverted (as it is closed) or is it in the 0.18
releases?

Peter.

Settings s = ImmutableSettings.settingsBuilder().put("cluster.name",
CLUSTER).build();
TransportClient client = new TransportClient(s);
client.addTransportAddress(new InetSocketTransportAddress("127.0.0.1",
PORT));
try {
client.admin().indices().create(new
CreateIndexRequest("tmp")).actionGet();} catch
(IndexAlreadyExistsException ex) {
}

client.prepareDeleteByQuery("tmp").setTypes("tweet").setQuery(QueryBuilders.matchAllQuery());

client.admin().indices().refresh(new RefreshRequest("tmp"));

BulkRequest br = new BulkRequest();
XContentBuilder cb =
JsonXContent.contentBuilder().startObject();
cb.field("dt", new Date(new Date().getTime() - 10000));
cb.field("text", "java");
br.add(new IndexRequest("tmp").id("1").type("tweet").source(cb));

cb = JsonXContent.contentBuilder().startObject();
cb.field("dt", new Date());
cb.field("text", "java");
br.add(new IndexRequest("tmp").id("2").type("tweet").source(cb));
client.bulk(br).actionGet();

client.admin().indices().refresh(new RefreshRequest("tmp"));
System.out.println("count:" + client.prepareCount("tmp").
execute().actionGet().count());
System.out.println("hits:" + client.prepareSearch("tmp").
setQuery(QueryBuilders.queryString("java").field("text")).
execute().actionGet().hits().totalHits());

System.out.println("hits sorted:" + client.prepareSearch("tmp").
setQuery(QueryBuilders.queryString("java").field("text")).
addSort("dt", SortOrder.DESC).
execute().actionGet().hits().totalHits());