Delete by query not working

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after 1h
but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.bertron@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after 1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1h' -d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d @query.json

Does it helps reproduce the error ?

Le 2013-05-13 à 18:13, David Pilato david@pilato.fr a écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.bertron@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after 1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey Loic,

the delete by query does not need a 'query' field in the JSON, as the whole
request body is the query itself, where as a search can consist of more
root level fields like 'query', 'filter' or 'facets'... So you need to send
a little bit different data to your delete by query request.

Hope this helps.

--Alex

On Tue, May 14, 2013 at 10:14 PM, Loïc Bertron loic.bertron@gmail.comwrote:

Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1h' -d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d @query.json

Does it helps reproduce the error ?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Le 2013-05-13 à 18:13, David Pilato david@pilato.fr a écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.bertron@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after 1h
but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey Alexander,

I tried this already, but no success.
Actually every time i'm running my delete query, i got this answer :
{"ok":true,"_indices":{"twitter":{"_shards":{"total":3,"successful":0,"failed":3}}}}

with this error in the logs :

[2013-05-15 09:37:05,386][DEBUG][action.deletebyquery ] [Damon Dran]
[twitter][0], node[nce4haLhTQeLaX76zJNPBg], [P], s[STARTED]: Failed to
execute [delete_by_query {[twitter][tweet], query [
"term" : { "user" : "kimchy" }
]}]
org.elasticsearch.index.query.QueryParsingException: [twitter] [_na] query
malformed, must start with start_object
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:170)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:268)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:216)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:370)
at
org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnPrimary(TransportShardDeleteByQueryAction.java:95)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

I'm using in that case the query from the doc : "term" : { "user" :
"kimchy" }

I'm also getting a lot of errors of this type : [2013-05-15
09:36:41,056][WARN ][transport.netty ] [Damon Dran] Message not
fully read (response) for [977] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@29a00473,
error [true], resetting

I have an Elasticsearch cluster with 2 nodes on 2 different locations
linked by a dedicated 100M link. I have only one index of 3 shards with 1
replica for each shard.

Do you have an idea ?

Le mercredi 15 mai 2013 02:35:23 UTC-4, Alexander Reelsen a écrit :

Hey Loic,

the delete by query does not need a 'query' field in the JSON, as the
whole request body is the query itself, where as a search can consist of
more root level fields like 'query', 'filter' or 'facets'... So you need to
send a little bit different data to your delete by query request.

Hope this helps.

--Alex

On Tue, May 14, 2013 at 10:14 PM, Loïc Bertron <loic.b...@gmail.com<javascript:>

wrote:

Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1h' -d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d @query.json

Does it helps reproduce the error ?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

Le 2013-05-13 à 18:13, David Pilato <da...@pilato.fr <javascript:>> a
écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron <loic.b...@gmail.com <javascript:>>
a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after
1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey Loic.

I made this GIST to illustrate how to build your query: Delete By Query usage · GitHub

Hope this helps

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 15 mai 2013 à 19:59, Loïc Bertron loic.bertron@gmail.com a écrit :

Hey Alexander,

I tried this already, but no success.
Actually every time i'm running my delete query, i got this answer :
{"ok":true,"_indices":{"twitter":{"_shards":{"total":3,"successful":0,"failed":3}}}}

with this error in the logs :

[2013-05-15 09:37:05,386][DEBUG][action.deletebyquery ] [Damon Dran] [twitter][0], node[nce4haLhTQeLaX76zJNPBg], [P], s[STARTED]: Failed to execute [delete_by_query {[twitter][tweet], query [
"term" : { "user" : "kimchy" }
]}]
org.elasticsearch.index.query.QueryParsingException: [twitter] [_na] query malformed, must start with start_object
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:170)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:268)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:216)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:370)
at org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnPrimary(TransportShardDeleteByQueryAction.java:95)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

I'm using in that case the query from the doc : "term" : { "user" : "kimchy" }

I'm also getting a lot of errors of this type : [2013-05-15 09:36:41,056][WARN ][transport.netty ] [Damon Dran] Message not fully read (response) for [977] handler org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@29a00473, error [true], resetting

I have an Elasticsearch cluster with 2 nodes on 2 different locations linked by a dedicated 100M link. I have only one index of 3 shards with 1 replica for each shard.

Do you have an idea ?

Le mercredi 15 mai 2013 02:35:23 UTC-4, Alexander Reelsen a écrit :
Hey Loic,

the delete by query does not need a 'query' field in the JSON, as the whole request body is the query itself, where as a search can consist of more root level fields like 'query', 'filter' or 'facets'... So you need to send a little bit different data to your delete by query request.

Hope this helps.

--Alex

On Tue, May 14, 2013 at 10:14 PM, Loïc Bertron loic.b...@gmail.com wrote:
Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1h' -d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d @query.json

Does it helps reproduce the error ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Le 2013-05-13 à 18:13, David Pilato da...@pilato.fr a écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.b...@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after 1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks David,

When i'm running your gist, it's working fine.
But when i re-run this on the tweets i'm indexing, i got some errors.

[2013-05-15 16:12:44,144][WARN ][transport.netty ] [Damon Dran]
Message not fully read (response) for [127080] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@4a9b1a42,
error [true], resetting

May 15 16:12:55 [WARNING] RemoteTransportException[Failed to deserialize
exception response from stream]; nested:
TransportSerializationException[Failed to deserialize exception response
from stream]; nested: StreamCorruptedException[unexpected end of block
data];

I can delete now using the query but I think the issue will appears again
in the future.

Do you know anything about theses errors ?

Le mercredi 15 mai 2013 15:58:50 UTC-4, David Pilato a écrit :

Hey Loic.

I made this GIST to illustrate how to build your query:
Delete By Query usage · GitHub

Hope this helps

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 15 mai 2013 à 19:59, Loïc Bertron <loic.b...@gmail.com <javascript:>>
a écrit :

Hey Alexander,

I tried this already, but no success.
Actually every time i'm running my delete query, i got this answer :

{"ok":true,"_indices":{"twitter":{"_shards":{"total":3,"successful":0,"failed":3}}}}

with this error in the logs :

[2013-05-15 09:37:05,386][DEBUG][action.deletebyquery ] [Damon Dran]
[twitter][0], node[nce4haLhTQeLaX76zJNPBg], [P], s[STARTED]: Failed to
execute [delete_by_query {[twitter][tweet], query [
"term" : { "user" : "kimchy" }
]}]
org.elasticsearch.index.query.QueryParsingException: [twitter] [_na] query
malformed, must start with start_object
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:170)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:268)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:216)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:370)
at
org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnPrimary(TransportShardDeleteByQueryAction.java:95)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

I'm using in that case the query from the doc : "term" : { "user" :
"kimchy" }

I'm also getting a lot of errors of this type : [2013-05-15
09:36:41,056][WARN ][transport.netty ] [Damon Dran] Message not
fully read (response) for [977] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@29a00473,
error [true], resetting

I have an Elasticsearch cluster with 2 nodes on 2 different locations
linked by a dedicated 100M link. I have only one index of 3 shards with 1
replica for each shard.

Do you have an idea ?

Le mercredi 15 mai 2013 02:35:23 UTC-4, Alexander Reelsen a écrit :

Hey Loic,

the delete by query does not need a 'query' field in the JSON, as the
whole request body is the query itself, where as a search can consist of
more root level fields like 'query', 'filter' or 'facets'... So you need to
send a little bit different data to your delete by query request.

Hope this helps.

--Alex

On Tue, May 14, 2013 at 10:14 PM, Loïc Bertron loic.b...@gmail.comwrote:

Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1h' -d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d
@query.json

Does it helps reproduce the error ?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Le 2013-05-13 à 18:13, David Pilato da...@pilato.fr a écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.b...@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after
1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sounds like you are mixing versions or something like that???
What is your architecture here?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 mai 2013 à 22:19, Loïc Bertron loic.bertron@gmail.com a écrit :

Thanks David,

When i'm running your gist, it's working fine.
But when i re-run this on the tweets i'm indexing, i got some errors.

[2013-05-15 16:12:44,144][WARN ][transport.netty ] [Damon Dran] Message not fully read (response) for [127080] handler org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@4a9b1a42, error [true], resetting

May 15 16:12:55 [WARNING] RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data];

I can delete now using the query but I think the issue will appears again in the future.

Do you know anything about theses errors ?

Le mercredi 15 mai 2013 15:58:50 UTC-4, David Pilato a écrit :

Hey Loic.

I made this GIST to illustrate how to build your query: Delete By Query usage · GitHub

Hope this helps

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 15 mai 2013 à 19:59, Loïc Bertron loic.b...@gmail.com a écrit :

Hey Alexander,

I tried this already, but no success.
Actually every time i'm running my delete query, i got this answer :
{"ok":true,"_indices":{"twitter":{"_shards":{"total":3,"successful":0,"failed":3}}}}

with this error in the logs :

[2013-05-15 09:37:05,386][DEBUG][action.deletebyquery ] [Damon Dran] [twitter][0], node[nce4haLhTQeLaX76zJNPBg], [P], s[STARTED]: Failed to execute [delete_by_query {[twitter][tweet], query [
"term" : { "user" : "kimchy" }
]}]
org.elasticsearch.index.query.QueryParsingException: [twitter] [_na] query malformed, must start with start_object
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:170)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:268)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:216)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:370)
at org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnPrimary(TransportShardDeleteByQueryAction.java:95)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

I'm using in that case the query from the doc : "term" : { "user" : "kimchy" }

I'm also getting a lot of errors of this type : [2013-05-15 09:36:41,056][WARN ][transport.netty ] [Damon Dran] Message not fully read (response) for [977] handler org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@29a00473, error [true], resetting

I have an Elasticsearch cluster with 2 nodes on 2 different locations linked by a dedicated 100M link. I have only one index of 3 shards with 1 replica for each shard.

Do you have an idea ?

Le mercredi 15 mai 2013 02:35:23 UTC-4, Alexander Reelsen a écrit :

Hey Loic,

the delete by query does not need a 'query' field in the JSON, as the whole request body is the query itself, where as a search can consist of more root level fields like 'query', 'filter' or 'facets'... So you need to send a little bit different data to your delete by query request.

Hope this helps.

--Alex

On Tue, May 14, 2013 at 10:14 PM, Loïc Bertron loic.b...@gmail.com wrote:

Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1h' -d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d @query.json

Does it helps reproduce the error ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Le 2013-05-13 à 18:13, David Pilato da...@pilato.fr a écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.b...@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after 1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey

minor difference in the docs, note the wrapping curly braces around the
"term" field. This also the reason why the exception message was, that
there is a missing START_OBJECT, which basically translates to a { in JSON
in order to build an object.

curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d '{
"term" : { "user" : "kimchy" }
}
'

Not sure where your other exceptions are coming from. Do you maybe use
different JVMs on your nodes? If so, try using the exact same JVM versions
on all nodes.

--Alex

On Wed, May 15, 2013 at 7:59 PM, Loïc Bertron loic.bertron@gmail.comwrote:

Hey Alexander,

I tried this already, but no success.
Actually every time i'm running my delete query, i got this answer :

{"ok":true,"_indices":{"twitter":{"_shards":{"total":3,"successful":0,"failed":3}}}}

with this error in the logs :

[2013-05-15 09:37:05,386][DEBUG][action.deletebyquery ] [Damon Dran]
[twitter][0], node[nce4haLhTQeLaX76zJNPBg], [P], s[STARTED]: Failed to
execute [delete_by_query {[twitter][tweet], query [
"term" : { "user" : "kimchy" }
]}]
org.elasticsearch.index.query.QueryParsingException: [twitter] [_na] query
malformed, must start with start_object
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:170)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:268)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:216)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:370)
at
org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnPrimary(TransportShardDeleteByQueryAction.java:95)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

I'm using in that case the query from the doc : "term" : { "user" :
"kimchy" }

I'm also getting a lot of errors of this type : [2013-05-15
09:36:41,056][WARN ][transport.netty ] [Damon Dran] Message not
fully read (response) for [977] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@29a00473,
error [true], resetting

I have an Elasticsearch cluster with 2 nodes on 2 different locations
linked by a dedicated 100M link. I have only one index of 3 shards with 1
replica for each shard.

Do you have an idea ?

Le mercredi 15 mai 2013 02:35:23 UTC-4, Alexander Reelsen a écrit :

Hey Loic,

the delete by query does not need a 'query' field in the JSON, as the
whole request body is the query itself, where as a search can consist of
more root level fields like 'query', 'filter' or 'facets'... So you need to
send a little bit different data to your delete by query request.

Hope this helps.

--Alex

On Tue, May 14, 2013 at 10:14 PM, Loïc Bertron loic.b...@gmail.comwrote:

Hey David,

Thanks for your reply.
Here is attached an example tweet that i'm inserting with this command :

curl -XPUT 'http://localhost:9200/**twitter/tweet/1?ttl=1h'<http://localhost:9200/twitter/tweet/1?ttl=1h'>-d @tweet.json

When i'm running the following command

curl -XGET 'http://localhost:9200/**twitter/tweet/_search'<http://localhost:9200/twitter/tweet/_search'>-d @query.json

I get more than 10000 results

If i try to delete by query with this one :

curl -XDELETE 'http://localhost:9200/**twitter/tweet/_query'<http://localhost:9200/twitter/tweet/_query'>-d @query.json

Does it helps reproduce the error ?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

Le 2013-05-13 à 18:13, David Pilato da...@pilato.fr a écrit :

Hey Loic!

Could you share a full curl recreation of your problem?
I would like to reproduce it.

Thanks

--
David Pilato | Technical Advocate | Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
|
* @scrutmydocs https://twitter.com/scrutmydocs

Le 13 mai 2013 à 23:16, Loïc Bertron loic.b...@gmail.com a écrit :

Hey guys,

I'm trying to delete some tweets i've been indexing automatically after
1h but even TTL or delete by query is not working.
When i set TTL, nothing happen.
Same result if i try this command :

curl -XDELETE localhost:9200/twitter/_query -d '{
"query" : {
"filtered": {
"query": {"match_all": {}},
"filter": {"range": {
"created_at": {
"lt": "now-1h"
}
}}
}
}
}'

What am i doing wrong ?

Loïc

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.