Creating Indicies - Sometimes necessary, other times not?

[James_Cook]

After using ES for over a year, I'm now trying to understand a long standing
problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an
embedded manner using Spring inside of a web application. I am using the
Java API to communicate with ES. There is only one instance of ES on my
system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested
in is sysinfo, and I want to use dynamic mappings so I have not created an
explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and see
the index module attempt to locate mappings in the data config path, and I
assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a
MissingIndexException.

I don't understand why an index succeeds when I perform a query before
indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by
programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
    final ClusterAdminClient cluster = client.admin().cluster();

cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

    ClusterStateResponse response =

cluster.prepareState().execute().actionGet();
boolean hasIndex = response.getState().metaData().hasIndex(esIndex);

    if(!hasIndex) {
        LOG.info("Index Not Found, creating index: " + esIndex);

client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but
my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc]
type[sysinfo] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[kimchy (Shay Banon)]

Heya,

An index operation will cause an index to be automatically created, a search on a specific index will not. Same applies to type, when indexing, the type is created, but, if you search/get a type before it is created, you will get TypeMissing failure (actually, in master, it will only happen for get, not for search).

Why not just create the index with empty mappings when the application starts up, and if its already there, just ignore the IndexAlreadyExists failure?

-shay.banon
On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an embedded manner using Spring inside of a web application. I am using the Java API to communicate with ES. There is only one instance of ES on my system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested in is sysinfo, and I want to use dynamic mappings so I have not created an explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and see the index module attempt to locate mappings in the data config path, and I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a MissingIndexException.

I don't understand why an index succeeds when I perform a query before indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
final ClusterAdminClient cluster = client.admin().cluster();
cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

ClusterStateResponse response = cluster.prepareState().execute().actionGet();
boolean hasIndex = response.getState().metaData().hasIndex(esIndex);

if(!hasIndex) {
LOG.info("Index Not Found, creating index: " + esIndex);
client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc] type[sysinfo] missing
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[James_Cook]

Thanks Shay. Is there a tech reason preventing the automatic creation of an
index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first
one I am using as a simple bucket for unstructured data and I want to use
dynamic mapping for it. I worked around my problem by creating a
sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Heya,

An index operation will cause an index to be automatically created, a
search on a specific index will not. Same applies to type, when indexing,
the type is created, but, if you search/get a type before it is created, you
will get TypeMissing failure (actually, in master, it will only happen for
get, not for search).

Why not just create the index with empty mappings when the application
starts up, and if its already there, just ignore the IndexAlreadyExists
failure?

-shay.banon

On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long
standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an
embedded manner using Spring inside of a web application. I am using the
Java API to communicate with ES. There is only one instance of ES on my
system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested
in is sysinfo, and I want to use dynamic mappings so I have not created an
explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and
see the index module attempt to locate mappings in the data config path, and
I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a
MissingIndexException.

I don't understand why an index succeeds when I perform a query before
indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by
programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
    final ClusterAdminClient cluster = client.admin().cluster();

cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

    ClusterStateResponse response =

cluster.prepareState().execute().actionGet();
boolean hasIndex =
response.getState().metaData().hasIndex(esIndex);

    if(!hasIndex) {
        LOG.info("Index Not Found, creating index: " + esIndex);

client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but
my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc]
type[sysinfo] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[kimchy (Shay Banon)]

It just does not seem correct to create an index / type when searching.
On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first one I am using as a simple bucket for unstructured data and I want to use dynamic mapping for it. I worked around my problem by creating a sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Heya,

An index operation will cause an index to be automatically created, a search on a specific index will not. Same applies to type, when indexing, the type is created, but, if you search/get a type before it is created, you will get TypeMissing failure (actually, in master, it will only happen for get, not for search).

Why not just create the index with empty mappings when the application starts up, and if its already there, just ignore the IndexAlreadyExists failure?

-shay.banon
On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an embedded manner using Spring inside of a web application. I am using the Java API to communicate with ES. There is only one instance of ES on my system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested in is sysinfo, and I want to use dynamic mappings so I have not created an explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and see the index module attempt to locate mappings in the data config path, and I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a MissingIndexException.

I don't understand why an index succeeds when I perform a query before indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
final ClusterAdminClient cluster = client.admin().cluster();
cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

ClusterStateResponse response = cluster.prepareState().execute().actionGet();
boolean hasIndex = response.getState().metaData().hasIndex(esIndex);

if(!hasIndex) {
LOG.info("Index Not Found, creating index: " + esIndex);
client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc] type[sysinfo] missing
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[James_Cook]

OK, but this "inconsistency" was hard for me to understand as a new user
(and a year later).

Works:
index(document)

Fails:
query(pred);

Works:
index(dummy);
query(pred);

-- jim

On Tue, Apr 19, 2011 at 7:37 PM, Shay Banon shay.banon@elasticsearch.comwrote:

It just does not seem correct to create an index / type when searching.

On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an
index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first
one I am using as a simple bucket for unstructured data and I want to use
dynamic mapping for it. I worked around my problem by creating a
sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Heya,

An index operation will cause an index to be automatically created, a
search on a specific index will not. Same applies to type, when indexing,
the type is created, but, if you search/get a type before it is created, you
will get TypeMissing failure (actually, in master, it will only happen for
get, not for search).

Why not just create the index with empty mappings when the application
starts up, and if its already there, just ignore the IndexAlreadyExists
failure?

-shay.banon

On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long
standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an
embedded manner using Spring inside of a web application. I am using the
Java API to communicate with ES. There is only one instance of ES on my
system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested
in is sysinfo, and I want to use dynamic mappings so I have not created an
explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and
see the index module attempt to locate mappings in the data config path, and
I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a
MissingIndexException.

I don't understand why an index succeeds when I perform a query before
indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by
programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
    final ClusterAdminClient cluster = client.admin().cluster();

cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

    ClusterStateResponse response =

cluster.prepareState().execute().actionGet();
boolean hasIndex =
response.getState().metaData().hasIndex(esIndex);

    if(!hasIndex) {
        LOG.info("Index Not Found, creating index: " + esIndex);

client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but
my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc]
type[sysinfo] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[kimchy (Shay Banon)]

I don't think so, at least not for me (the difficult to understand part). If you want full consistency, you can disable index creation on index API, then it will be consistent. I don't think that an index / type should be created when searching.
On Wednesday, April 20, 2011 at 4:16 AM, James Cook wrote:

OK, but this "inconsistency" was hard for me to understand as a new user (and a year later).

Works:
index(document)

Fails:
query(pred);

Works:
index(dummy);
query(pred);

-- jim

On Tue, Apr 19, 2011 at 7:37 PM, Shay Banon shay.banon@elasticsearch.com wrote:

It just does not seem correct to create an index / type when searching.
On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first one I am using as a simple bucket for unstructured data and I want to use dynamic mapping for it. I worked around my problem by creating a sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Heya,

An index operation will cause an index to be automatically created, a search on a specific index will not. Same applies to type, when indexing, the type is created, but, if you search/get a type before it is created, you will get TypeMissing failure (actually, in master, it will only happen for get, not for search).

Why not just create the index with empty mappings when the application starts up, and if its already there, just ignore the IndexAlreadyExists failure?

-shay.banon
On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an embedded manner using Spring inside of a web application. I am using the Java API to communicate with ES. There is only one instance of ES on my system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested in is sysinfo, and I want to use dynamic mappings so I have not created an explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and see the index module attempt to locate mappings in the data config path, and I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a MissingIndexException.

I don't understand why an index succeeds when I perform a query before indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
final ClusterAdminClient cluster = client.admin().cluster();
cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

ClusterStateResponse response = cluster.prepareState().execute().actionGet();
boolean hasIndex = response.getState().metaData().hasIndex(esIndex);

if(!hasIndex) {
LOG.info("Index Not Found, creating index: " + esIndex);
client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc] type[sysinfo] missing
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[James_Cook]

I'm starting to feel you really don't think it is a good idea to create an
index when searching.

What would you think about adding a configuration option for a comma
separated list of indexes that should be created at startup?

-- jim

On Tue, Apr 19, 2011 at 9:18 PM, Shay Banon shay.banon@elasticsearch.comwrote:

I don't think so, at least not for me (the difficult to understand part).
If you want full consistency, you can disable index creation on index API,
then it will be consistent. I don't think that an index / type should be
created when searching.

On Wednesday, April 20, 2011 at 4:16 AM, James Cook wrote:

OK, but this "inconsistency" was hard for me to understand as a new user
(and a year later).

Works:
index(document)

Fails:
query(pred);

Works:
index(dummy);
query(pred);

-- jim

On Tue, Apr 19, 2011 at 7:37 PM, Shay Banon shay.banon@elasticsearch.comwrote:

It just does not seem correct to create an index / type when searching.

On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an
index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first
one I am using as a simple bucket for unstructured data and I want to use
dynamic mapping for it. I worked around my problem by creating a
sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Heya,

An index operation will cause an index to be automatically created, a
search on a specific index will not. Same applies to type, when indexing,
the type is created, but, if you search/get a type before it is created, you
will get TypeMissing failure (actually, in master, it will only happen for
get, not for search).

Why not just create the index with empty mappings when the application
starts up, and if its already there, just ignore the IndexAlreadyExists
failure?

-shay.banon

On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long
standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an
embedded manner using Spring inside of a web application. I am using the
Java API to communicate with ES. There is only one instance of ES on my
system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested
in is sysinfo, and I want to use dynamic mappings so I have not created an
explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and
see the index module attempt to locate mappings in the data config path, and
I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a
MissingIndexException.

I don't understand why an index succeeds when I perform a query before
indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by
programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
    final ClusterAdminClient cluster = client.admin().cluster();

cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

    ClusterStateResponse response =

cluster.prepareState().execute().actionGet();
boolean hasIndex =
response.getState().metaData().hasIndex(esIndex);

    if(!hasIndex) {
        LOG.info("Index Not Found, creating index: " + esIndex);

client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but
my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc]
type[sysinfo] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[kimchy (Shay Banon)]

The problem with startup parameters is that they don't lend themselves well to distributed systems. A more proper solution is to just use the create index API when starting the node, making sure things are there. Its pretty simple to do, no?
On Wednesday, April 20, 2011 at 4:23 AM, James Cook wrote:

I'm starting to feel you really don't think it is a good idea to create an index when searching.

What would you think about adding a configuration option for a comma separated list of indexes that should be created at startup?

-- jim

On Tue, Apr 19, 2011 at 9:18 PM, Shay Banon shay.banon@elasticsearch.com wrote:

I don't think so, at least not for me (the difficult to understand part). If you want full consistency, you can disable index creation on index API, then it will be consistent. I don't think that an index / type should be created when searching.
On Wednesday, April 20, 2011 at 4:16 AM, James Cook wrote:

OK, but this "inconsistency" was hard for me to understand as a new user (and a year later).

Works:
index(document)

Fails:
query(pred);

Works:
index(dummy);
query(pred);

-- jim

On Tue, Apr 19, 2011 at 7:37 PM, Shay Banon shay.banon@elasticsearch.com wrote:

It just does not seem correct to create an index / type when searching.
On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first one I am using as a simple bucket for unstructured data and I want to use dynamic mapping for it. I worked around my problem by creating a sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Heya,

An index operation will cause an index to be automatically created, a search on a specific index will not. Same applies to type, when indexing, the type is created, but, if you search/get a type before it is created, you will get TypeMissing failure (actually, in master, it will only happen for get, not for search).

Why not just create the index with empty mappings when the application starts up, and if its already there, just ignore the IndexAlreadyExists failure?

-shay.banon
On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an embedded manner using Spring inside of a web application. I am using the Java API to communicate with ES. There is only one instance of ES on my system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested in is sysinfo, and I want to use dynamic mappings so I have not created an explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and see the index module attempt to locate mappings in the data config path, and I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a MissingIndexException.

I don't understand why an index succeeds when I perform a query before indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
final ClusterAdminClient cluster = client.admin().cluster();
cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

ClusterStateResponse response = cluster.prepareState().execute().actionGet();
boolean hasIndex = response.getState().metaData().hasIndex(esIndex);

if(!hasIndex) {
LOG.info("Index Not Found, creating index: " + esIndex);
client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc] type[sysinfo] missing
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[James_Cook]

Sure, no problem. Thanks for the quick responses.

-- jim

On Tue, Apr 19, 2011 at 9:25 PM, Shay Banon shay.banon@elasticsearch.comwrote:

The problem with startup parameters is that they don't lend themselves
well to distributed systems. A more proper solution is to just use the
create index API when starting the node, making sure things are there. Its
pretty simple to do, no?

On Wednesday, April 20, 2011 at 4:23 AM, James Cook wrote:

I'm starting to feel you really don't think it is a good idea to create an
index when searching.

What would you think about adding a configuration option for a comma
separated list of indexes that should be created at startup?

-- jim

On Tue, Apr 19, 2011 at 9:18 PM, Shay Banon shay.banon@elasticsearch.comwrote:

I don't think so, at least not for me (the difficult to understand
part). If you want full consistency, you can disable index creation on index
API, then it will be consistent. I don't think that an index / type should
be created when searching.

On Wednesday, April 20, 2011 at 4:16 AM, James Cook wrote:

OK, but this "inconsistency" was hard for me to understand as a new user
(and a year later).

Works:
index(document)

Fails:
query(pred);

Works:
index(dummy);
query(pred);

-- jim

On Tue, Apr 19, 2011 at 7:37 PM, Shay Banon shay.banon@elasticsearch.comwrote:

It just does not seem correct to create an index / type when searching.

On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an
index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first
one I am using as a simple bucket for unstructured data and I want to use
dynamic mapping for it. I worked around my problem by creating a
sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Heya,

An index operation will cause an index to be automatically created, a
search on a specific index will not. Same applies to type, when indexing,
the type is created, but, if you search/get a type before it is created, you
will get TypeMissing failure (actually, in master, it will only happen for
get, not for search).

Why not just create the index with empty mappings when the application
starts up, and if its already there, just ignore the IndexAlreadyExists
failure?

-shay.banon

On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long
standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an
embedded manner using Spring inside of a web application. I am using the
Java API to communicate with ES. There is only one instance of ES on my
system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested
in is sysinfo, and I want to use dynamic mappings so I have not created an
explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and
see the index module attempt to locate mappings in the data config path, and
I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a
MissingIndexException.

I don't understand why an index succeeds when I perform a query before
indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by
programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
    final ClusterAdminClient cluster = client.admin().cluster();

cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

    ClusterStateResponse response =

cluster.prepareState().execute().actionGet();
boolean hasIndex =
response.getState().metaData().hasIndex(esIndex);

    if(!hasIndex) {
        LOG.info("Index Not Found, creating index: " + esIndex);

client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but
my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc]
type[sysinfo] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?

[Karussell1 (Karussell)]

You can always do this by hand. Check if an index exist, if not create
it. Or like so in Java:

public void saveCreateIndex(String indexName) {
    try {
      client.admin().indices().create(new

CreateIndexRequest(indexName)).actionGet();
} catch (Exception ex) {
// already exists
}
}

Regards,
Peter.

On Apr 20, 3:33 am, James Cook jc...@tracermedia.com wrote:

Sure, no problem. Thanks for the quick responses.

-- jim

On Tue, Apr 19, 2011 at 9:25 PM, Shay Banon shay.ba...@elasticsearch.comwrote:

The problem with startup parameters is that they don't lend themselves
well to distributed systems. A more proper solution is to just use the
create index API when starting the node, making sure things are there. Its
pretty simple to do, no?

On Wednesday, April 20, 2011 at 4:23 AM, James Cook wrote:

I'm starting to feel you really don't think it is a good idea to create an
index when searching.

What would you think about adding a configuration option for a comma
separated list of indexes that should be created at startup?

-- jim

On Tue, Apr 19, 2011 at 9:18 PM, Shay Banon shay.ba...@elasticsearch.comwrote:

I don't think so, at least not for me (the difficult to understand
part). If you want full consistency, you can disable index creation on index
API, then it will be consistent. I don't think that an index / type should
be created when searching.

On Wednesday, April 20, 2011 at 4:16 AM, James Cook wrote:

OK, but this "inconsistency" was hard for me to understand as a new user
(and a year later).

Works:
index(document)

Fails:
query(pred);

Works:
index(dummy);
query(pred);

-- jim

On Tue, Apr 19, 2011 at 7:37 PM, Shay Banon shay.ba...@elasticsearch.comwrote:

It just does not seem correct to create an index / type when searching.

On Wednesday, April 20, 2011 at 2:34 AM, James Cook wrote:

Thanks Shay. Is there a tech reason preventing the automatic creation of an
index whether it is accessed for read or write?

I have other types which use mapping files. This sysinfo type is the first
one I am using as a simple bucket for unstructured data and I want to use
dynamic mapping for it. I worked around my problem by creating a
sysinfo.json mapping file that looks like this:

{ "sysinfo" : { } }

Thanks,
jim

On Tue, Apr 19, 2011 at 7:23 PM, Shay Banon shay.ba...@elasticsearch.comwrote:

Heya,

An index operation will cause an index to be automatically created, a
search on a specific index will not. Same applies to type, when indexing,
the type is created, but, if you search/get a type before it is created, you
will get TypeMissing failure (actually, in master, it will only happen for
get, not for search).

Why not just create the index with empty mappings when the application
starts up, and if its already there, just ignore the IndexAlreadyExists
failure?

-shay.banon

On Wednesday, April 20, 2011 at 2:14 AM, James Cook wrote:

After using ES for over a year, I'm now trying to understand a long
standing problem we experience.

Assume I have no gateway configured and I have started ES 0.15.2 in an
embedded manner using Spring inside of a web application. I am using the
Java API to communicate with ES. There is only one instance of ES on my
system, so no clustering.

I have a single index (ppkc) with multiple types. The type I am interested
in is sysinfo, and I want to use dynamic mappings so I have not created an
explicit mapping file.

Scenario #1

If I attempt to index a document, I can turn on debug log statements and
see the index module attempt to locate mappings in the data config path, and
I assume since none are found the default dynamic mapping kicks in.

Any subsequent queries on the 'sysinfo' type return as expected.

Scenario #2

However, if I attempt to query for a document before indexing, I get a
MissingIndexException.

I don't understand why an index succeeds when I perform a query before
indexing a document. If I do the index first then a query, all is well.

So, thinking that I can catch the exception and handle it by
programmatically forcing an index I wrote this function:

private void checkForIndex(Client client, String esIndex) {
    final ClusterAdminClient cluster = client.admin().cluster();

cluster.prepareHealth().setWaitForYellowStatus().execute().actionGet();

    ClusterStateResponse response =

cluster.prepareState().execute().actionGet();
boolean hasIndex =
response.getState().metaData().hasIndex(esIndex);

    if(!hasIndex) {
        LOG.info("Index Not Found, creating index: " + esIndex);

client.admin().indices().create(createIndexRequest(esIndex)).actionGet();
}
}

However, this doesn't work either. The index is created without issue, but
my query results in this exception:

Caused by: org.elasticsearch.indices.TypeMissingException: [ppkc]
type[sysinfo] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:62)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:250)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:239)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:238)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Any thoughts?