Inconsistent results with space'd values over TCP vs. HTTP

Manish_Pandit · October 25, 2011, 10:59pm

Hi there,

Here is the query that we're sending to an Elastic Search Index via
the Java Driver, TCP Port 9300.

{
"from" : 0,
"size" : 20,
"query" : {
"match_all" : {
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"metadata.prime" : false
}
}, {
"range" : {
"metadata.publishDate" : {
"from" : 1288047197219,
"to" : 1319583197219,
"include_lower" : true,
"include_upper" : false
}
}
}, {
"or" : {
"filters" : [ {
"term" : {
"metadata.state" : "published"
}
} ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"metadata.name" : "call of duty"
}
} ]
}
} ]
}
},
"fields" : [ ],
"sort" : [ {
"metadata.publishDate" : {
"order" : "desc"
}
} ]
}

This query returns Zero hits. At the same time, when I submit this
query (via HTTP GET on port 9200) on the _search endpoint of that
index, I get results as expected.

This is happening when any of the values have spaces in them (please
note metadata.name as "call of duty"). If the values do not have
spaces, the queries sent through the TCP Connection work just as good
as them over HTTP.

This has put us in a fix - should we send all queries over HTTP and
then parse the results (much cumbersome), or is there something we are
missing while we are sending this query through the Java Driver as

   val result = query.execute.actionGet()

Please let me know. I'd appreciate any hints or pointers.

-cheers,
Manish

kimchy · October 26, 2011, 3:12am

Thats strange..., how are you executing the search request using the Java
API? The Java API ends up building the same query structure as the rest API.

On Wed, Oct 26, 2011 at 12:59 AM, Manish Pandit pandit.manish@gmail.comwrote:

Hi there,

Here is the query that we're sending to an Elastic Search Index via
the Java Driver, TCP Port 9300.

{
"from" : 0,
"size" : 20,
"query" : {
"match_all" : {
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"metadata.prime" : false
}
}, {
"range" : {
"metadata.publishDate" : {
"from" : 1288047197219,
"to" : 1319583197219,
"include_lower" : true,
"include_upper" : false
}
}
}, {
"or" : {
"filters" : [ {
"term" : {
"metadata.state" : "published"
}
} ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"metadata.name" : "call of duty"
}
} ]
}
} ]
}
},
"fields" : ,
"sort" : [ {
"metadata.publishDate" : {
"order" : "desc"
}
} ]
}

This query returns Zero hits. At the same time, when I submit this
query (via HTTP GET on port 9200) on the _search endpoint of that
index, I get results as expected.

This is happening when any of the values have spaces in them (please
note metadata.name as "call of duty"). If the values do not have
spaces, the queries sent through the TCP Connection work just as good
as them over HTTP.

This has put us in a fix - should we send all queries over HTTP and
then parse the results (much cumbersome), or is there something we are
missing while we are sending this query through the Java Driver as
  val result = query.execute.actionGet()
Please let me know. I'd appreciate any hints or pointers.

-cheers,
Manish

Manish_Pandit · October 26, 2011, 4:15am

On Oct 25, 8:12 pm, Shay Banon kim...@gmail.com wrote:

Thats strange..., how are you executing the search request using the Java
API? The Java API ends up building the same query structure as the rest API.

On Wed, Oct 26, 2011 at 12:59 AM, Manish Pandit pandit.man...@gmail.comwrote:

Hi there,

Here is the query that we're sending to an Elastic Search Index via
the Java Driver, TCP Port 9300.

{
"from" : 0,
"size" : 20,
"query" : {
"match_all" : {
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"metadata.prime" : false
}
}, {
"range" : {
"metadata.publishDate" : {
"from" : 1288047197219,
"to" : 1319583197219,
"include_lower" : true,
"include_upper" : false
}
}
}, {
"or" : {
"filters" : [ {
"term" : {
"metadata.state" : "published"
}
} ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"metadata.name" : "call of duty"
}
} ]
}
} ]
}
},
"fields" : ,
"sort" : [ {
"metadata.publishDate" : {
"order" : "desc"
}
} ]
}

This query returns Zero hits. At the same time, when I submit this
query (via HTTP GET on port 9200) on the _search endpoint of that
index, I get results as expected.

This is happening when any of the values have spaces in them (please
note metadata.name as "call of duty"). If the values do not have
spaces, the queries sent through the TCP Connection work just as good
as them over HTTP.

This has put us in a fix - should we send all queries over HTTP and
then parse the results (much cumbersome), or is there something we are
missing while we are sending this query through the Java Driver as
  val result = query.execute.actionGet()
Please let me know. I'd appreciate any hints or pointers.

-cheers,
Manish

Hi Shay - Thanks for your reply

Exactly! I am building the query using the SearchBuilder - like this:

val query = new

SearchRequestBuilder(cf.elasticClient).setQuery(matchAllQuery()).

setFrom(startIndex).setSize(count).setNoFields().setIndices(cf.searchConfiguration.indexName).
setTypes(cf.searchConfiguration.videoModel).addSort(sort,
if (sortOrder == "desc") SortOrder.DESC else SortOrder.ASC)
query.setFilter(superFilter)

Where I am adding more filters (as you can see from the output). At
the end I log the query variable, which renders the JSON that I pasted
above.

Next I do this :

    val result = query.execute.actionGet()

Which returns me no results. However, when I paste the JSON produced
as the value of query, (on the server:9200/index/type/_search) I am
able to see the results. This is only happening when any of the values
contains a space. If there are no spaces, it works great via the Java
driver as well as HTTP.

Now I am trying to use Apache Commons HTTP Client to send the same
JSON over HTTP, but the client libraries do not support sending a body
with GET request. Sending the same on POST does not work obviously.
Next attempt would be to use Java's HttpURLConnection and hopefully
that'd let me send a payload over GET but if that too doesn't work, I
really am out of options

kimchy · October 26, 2011, 4:21am

I think that there is something wrong with how you execute the search many
using the Java API, I am still missing the complete picture. Obviously,
something is strange here. If you can creata a simple test program, that
also initializes the index with some data, to show the failure, I will be
happy to look at it.

Regarding using the REST API, you can use POST as well to execute search,
not just GET. But, I think that you are giving up on the Java API too
fast...

On Wed, Oct 26, 2011 at 6:15 AM, Manish Pandit pandit.manish@gmail.comwrote:

On Oct 25, 8:12 pm, Shay Banon kim...@gmail.com wrote:
Thats strange..., how are you executing the search request using the Java
API? The Java API ends up building the same query structure as the rest
API.

On Wed, Oct 26, 2011 at 12:59 AM, Manish Pandit <pandit.man...@gmail.com
wrote:

Hi there,

Here is the query that we're sending to an Elastic Search Index via
the Java Driver, TCP Port 9300.

{
"from" : 0,
"size" : 20,
"query" : {
"match_all" : {
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"metadata.prime" : false
}
}, {
"range" : {
"metadata.publishDate" : {
"from" : 1288047197219,
"to" : 1319583197219,
"include_lower" : true,
"include_upper" : false
}
}
}, {
"or" : {
"filters" : [ {
"term" : {
"metadata.state" : "published"
}
} ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"metadata.name" : "call of duty"
}
} ]
}
} ]
}
},
"fields" : ,
"sort" : [ {
"metadata.publishDate" : {
"order" : "desc"
}
} ]
}

This query returns Zero hits. At the same time, when I submit this
query (via HTTP GET on port 9200) on the _search endpoint of that
index, I get results as expected.

This is happening when any of the values have spaces in them (please
note metadata.name as "call of duty"). If the values do not have
spaces, the queries sent through the TCP Connection work just as good
as them over HTTP.

This has put us in a fix - should we send all queries over HTTP and
then parse the results (much cumbersome), or is there something we are
missing while we are sending this query through the Java Driver as
  val result = query.execute.actionGet()
Please let me know. I'd appreciate any hints or pointers.

-cheers,
Manish
Hi Shay - Thanks for your reply

Exactly! I am building the query using the SearchBuilder - like this:

val query = new
SearchRequestBuilder(cf.elasticClient).setQuery(matchAllQuery()).

setFrom(startIndex).setSize(count).setNoFields().setIndices(cf.searchConfiguration.indexName).
setTypes(cf.searchConfiguration.videoModel).addSort(sort,
if (sortOrder == "desc") SortOrder.DESC else SortOrder.ASC)
query.setFilter(superFilter)

Where I am adding more filters (as you can see from the output). At
the end I log the query variable, which renders the JSON that I pasted
above.

Next I do this :
   val result = query.execute.actionGet()
Which returns me no results. However, when I paste the JSON produced
as the value of query, (on the server:9200/index/type/_search) I am
able to see the results. This is only happening when any of the values
contains a space. If there are no spaces, it works great via the Java
driver as well as HTTP.

Now I am trying to use Apache Commons HTTP Client to send the same
JSON over HTTP, but the client libraries do not support sending a body
with GET request. Sending the same on POST does not work obviously.
Next attempt would be to use Java's HttpURLConnection and hopefully
that'd let me send a payload over GET but if that too doesn't work, I
really am out of options

Manish_Pandit · October 26, 2011, 4:24am

On Oct 25, 9:21 pm, Shay Banon kim...@gmail.com wrote:

I think that there is something wrong with how you execute the search many
using the Java API, I am still missing the complete picture. Obviously,
something is strange here. If you can creata a simple test program, that
also initializes the index with some data, to show the failure, I will be
happy to look at it.

Regarding using the REST API, you can use POST as well to execute search,
not just GET. But, I think that you are giving up on the Java API too
fast...

On Wed, Oct 26, 2011 at 6:15 AM, Manish Pandit pandit.man...@gmail.comwrote:

On Oct 25, 8:12 pm, Shay Banon kim...@gmail.com wrote:

Thats strange..., how are you executing the search request using the Java
API? The Java API ends up building the same query structure as the rest
API.

On Wed, Oct 26, 2011 at 12:59 AM, Manish Pandit <pandit.man...@gmail.com
wrote:

Hi there,

Here is the query that we're sending to an Elastic Search Index via
the Java Driver, TCP Port 9300.

{
"from" : 0,
"size" : 20,
"query" : {
"match_all" : {
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"metadata.prime" : false
}
}, {
"range" : {
"metadata.publishDate" : {
"from" : 1288047197219,
"to" : 1319583197219,
"include_lower" : true,
"include_upper" : false
}
}
}, {
"or" : {
"filters" : [ {
"term" : {
"metadata.state" : "published"
}
} ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"metadata.name" : "call of duty"
}
} ]
}
} ]
}
},
"fields" : ,
"sort" : [ {
"metadata.publishDate" : {
"order" : "desc"
}
} ]
}

This query returns Zero hits. At the same time, when I submit this
query (via HTTP GET on port 9200) on the _search endpoint of that
index, I get results as expected.

This is happening when any of the values have spaces in them (please
note metadata.name as "call of duty"). If the values do not have
spaces, the queries sent through the TCP Connection work just as good
as them over HTTP.

This has put us in a fix - should we send all queries over HTTP and
then parse the results (much cumbersome), or is there something we are
missing while we are sending this query through the Java Driver as
  val result = query.execute.actionGet()
Please let me know. I'd appreciate any hints or pointers.

-cheers,
Manish

Hi Shay - Thanks for your reply

Exactly! I am building the query using the SearchBuilder - like this:

val query = new
SearchRequestBuilder(cf.elasticClient).setQuery(matchAllQuery()).

setFrom(startIndex).setSize(count).setNoFields().setIndices(cf.searchConfig uration.indexName).
setTypes(cf.searchConfiguration.videoModel).addSort(sort,
if (sortOrder == "desc") SortOrder.DESC else SortOrder.ASC)
query.setFilter(superFilter)

Where I am adding more filters (as you can see from the output). At
the end I log the query variable, which renders the JSON that I pasted
above.

Next I do this :
   val result = query.execute.actionGet()
Which returns me no results. However, when I paste the JSON produced
as the value of query, (on the server:9200/index/type/_search) I am
able to see the results. This is only happening when any of the values
contains a space. If there are no spaces, it works great via the Java
driver as well as HTTP.

Now I am trying to use Apache Commons HTTP Client to send the same
JSON over HTTP, but the client libraries do not support sending a body
with GET request. Sending the same on POST does not work obviously.
Next attempt would be to use Java's HttpURLConnection and hopefully
that'd let me send a payload over GET but if that too doesn't work, I
really am out of options

Hi Shay - Let me take another look at what is going on with my Java
Client. I'll probably put a test code with mapping file etc. on my
private repo on github and give you access to it if that helps.

Thanks again - We plan to put Elastic in Production as early as next
month, and except for this everything else has been sailing super
smooth

-cheers,
Manish

Manish_Pandit · October 26, 2011, 11:31pm

On Oct 25, 9:24 pm, Manish Pandit pandit.man...@gmail.com wrote:

On Oct 25, 9:21 pm, Shay Banon kim...@gmail.com wrote:

I think that there is something wrong with how you execute the search many
using the Java API, I am still missing the complete picture. Obviously,
something is strange here. If you can creata a simple test program, that
also initializes the index with some data, to show the failure, I will be
happy to look at it.

Regarding using the REST API, you can use POST as well to execute search,
not just GET. But, I think that you are giving up on the Java API too
fast...

On Wed, Oct 26, 2011 at 6:15 AM, Manish Pandit pandit.man...@gmail.comwrote:

On Oct 25, 8:12 pm, Shay Banon kim...@gmail.com wrote:

Thats strange..., how are you executing the search request using the Java
API? The Java API ends up building the same query structure as the rest
API.

On Wed, Oct 26, 2011 at 12:59 AM, Manish Pandit <pandit.man...@gmail.com
wrote:

Hi there,

Here is the query that we're sending to an Elastic Search Index via
the Java Driver, TCP Port 9300.

{
"from" : 0,
"size" : 20,
"query" : {
"match_all" : {
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"metadata.prime" : false
}
}, {
"range" : {
"metadata.publishDate" : {
"from" : 1288047197219,
"to" : 1319583197219,
"include_lower" : true,
"include_upper" : false
}
}
}, {
"or" : {
"filters" : [ {
"term" : {
"metadata.state" : "published"
}
} ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"metadata.name" : "call of duty"
}
} ]
}
} ]
}
},
"fields" : ,
"sort" : [ {
"metadata.publishDate" : {
"order" : "desc"
}
} ]
}

This query returns Zero hits. At the same time, when I submit this
query (via HTTP GET on port 9200) on the _search endpoint of that
index, I get results as expected.

This is happening when any of the values have spaces in them (please
note metadata.name as "call of duty"). If the values do not have
spaces, the queries sent through the TCP Connection work just as good
as them over HTTP.

This has put us in a fix - should we send all queries over HTTP and
then parse the results (much cumbersome), or is there something we are
missing while we are sending this query through the Java Driver as
  val result = query.execute.actionGet()
Please let me know. I'd appreciate any hints or pointers.

-cheers,
Manish

Hi Shay - Thanks for your reply

Exactly! I am building the query using the SearchBuilder - like this:

val query = new
SearchRequestBuilder(cf.elasticClient).setQuery(matchAllQuery()).

setFrom(startIndex).setSize(count).setNoFields().setIndices(cf.searchConfig uration.indexName).
setTypes(cf.searchConfiguration.videoModel).addSort(sort,
if (sortOrder == "desc") SortOrder.DESC else SortOrder.ASC)
query.setFilter(superFilter)

Where I am adding more filters (as you can see from the output). At
the end I log the query variable, which renders the JSON that I pasted
above.

Next I do this :
   val result = query.execute.actionGet()
Which returns me no results. However, when I paste the JSON produced
as the value of query, (on the server:9200/index/type/_search) I am
able to see the results. This is only happening when any of the values
contains a space. If there are no spaces, it works great via the Java
driver as well as HTTP.

Now I am trying to use Apache Commons HTTP Client to send the same
JSON over HTTP, but the client libraries do not support sending a body
with GET request. Sending the same on POST does not work obviously.
Next attempt would be to use Java's HttpURLConnection and hopefully
that'd let me send a payload over GET but if that too doesn't work, I
really am out of options

Hi Shay - Let me take another look at what is going on with my Java
Client. I'll probably put a test code with mapping file etc. on my
private repo on github and give you access to it if that helps.

Thanks again - We plan to put Elastic in Production as early as next
month, and except for this everything else has been sailing super
smooth

-cheers,
Manish

Hi Shay - My bad. I had the date format wrong, and should have been
using wildcardQuery. This is resolved now - thanks so much for your
replies.

-cheers,
Manish