Newbie query question

Elastic Stack Elasticsearch
1

Hi all:

Just getting started with ElasticSearch. I've taken some log4j output
and inserted it into ElasticSearch to give me some data to play with,
which looks like the following:

{
total: 34,
max_score: 1,
hits: [
{
_index: "tomcat",
_type: "base",
_id: "an_id",
_score: 1,
_source: {
loggerName: "stuff",
message: "stuff",
level: "stuff",
timestamp: "stuff",
thread: "stuff"
}
},

So, now I'm trying to actually query / sort this data. I'm building a
JSON request that looks like:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level" : { "order" : "asc" }
],
"query" : {
"term" : { "level" : targetLevel }
}
}

And execute the request with:

$.getJSON(myUrl, q, function(rData) {
console.log(rData);
});

However, what I get back is an unordered list of every log item,
rather than the sorted list that I'm looking for.

I've also tried:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level"
],
"query" : {
"term" : { "level" : targetLevel }
}
}

and a number of other variations, but haven't had much success.
Neither the search nor the sort seem to work.

I'm sure I must be making an obvious mistake. That said, if anyone
could point me in the right direction here, it would be greatly
appreciated :slight_smile:

Also, related question: is there a simple way to ask ElasticSearch to
be verbose about how / why it's filtering and sorting its results?

Thanks all!

--

2

Not sure if it's your concern, but there is this part of documentation about sorting on string values:

When sorting, the relevant sorted field values are loaded into memory. This means that per shard, there should be enough memory to contain them. For string based types, the field sorted on should not be analyzed / tokenized. For numeric types, if possible, it is recommended to explicitly set the type to six_hun types (like short,integer and float).

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2012 à 17:52, Steve cubic1271@gmail.com a écrit :

Hi all:

Just getting started with Elasticsearch. I've taken some log4j output
and inserted it into Elasticsearch to give me some data to play with,
which looks like the following:

{
total: 34,
max_score: 1,
hits: [
{
_index: "tomcat",
_type: "base",
_id: "an_id",
_score: 1,
_source: {
loggerName: "stuff",
message: "stuff",
level: "stuff",
timestamp: "stuff",
thread: "stuff"
}
},

So, now I'm trying to actually query / sort this data. I'm building a
JSON request that looks like:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level" : { "order" : "asc" }
],
"query" : {
"term" : { "level" : targetLevel }
}
}

And execute the request with:

$.getJSON(myUrl, q, function(rData) {
console.log(rData);
});

However, what I get back is an unordered list of every log item,
rather than the sorted list that I'm looking for.

I've also tried:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level"
],
"query" : {
"term" : { "level" : targetLevel }
}
}

and a number of other variations, but haven't had much success.
Neither the search nor the sort seem to work.

I'm sure I must be making an obvious mistake. That said, if anyone
could point me in the right direction here, it would be greatly
appreciated :slight_smile:

Also, related question: is there a simple way to ask Elasticsearch to
be verbose about how / why it's filtering and sorting its results?

Thanks all!

--

--

3

for the second part of your question, just add "explain":true to your
json object. You'll get more information than you asked for, but it
may give you some clues.

On Thu, Oct 4, 2012 at 8:52 AM, Steve cubic1271@gmail.com wrote:

Hi all:

Just getting started with Elasticsearch. I've taken some log4j output
and inserted it into Elasticsearch to give me some data to play with,
which looks like the following:

{
total: 34,
max_score: 1,
hits: [
{
_index: "tomcat",
_type: "base",
_id: "an_id",
_score: 1,
_source: {
loggerName: "stuff",
message: "stuff",
level: "stuff",
timestamp: "stuff",
thread: "stuff"
}
},

So, now I'm trying to actually query / sort this data. I'm building a
JSON request that looks like:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level" : { "order" : "asc" }
],
"query" : {
"term" : { "level" : targetLevel }
}
}

And execute the request with:

$.getJSON(myUrl, q, function(rData) {
console.log(rData);
});

However, what I get back is an unordered list of every log item,
rather than the sorted list that I'm looking for.

I've also tried:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level"
],
"query" : {
"term" : { "level" : targetLevel }
}
}

and a number of other variations, but haven't had much success.
Neither the search nor the sort seem to work.

I'm sure I must be making an obvious mistake. That said, if anyone
could point me in the right direction here, it would be greatly
appreciated :slight_smile:

Also, related question: is there a simple way to ask Elasticsearch to
be verbose about how / why it's filtering and sorting its results?

Thanks all!

--

--

4

Explain will give you clues on how score is computed. IMHO, you won't get details why some results are filtered or not.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2012 à 18:07, Raffaele Sena raff367@gmail.com a écrit :

for the second part of your question, just add "explain":true to your
json object. You'll get more information than you asked for, but it
may give you some clues.

On Thu, Oct 4, 2012 at 8:52 AM, Steve cubic1271@gmail.com wrote:

Hi all:

Just getting started with Elasticsearch. I've taken some log4j output
and inserted it into Elasticsearch to give me some data to play with,
which looks like the following:

{
total: 34,
max_score: 1,
hits: [
{
_index: "tomcat",
_type: "base",
_id: "an_id",
_score: 1,
_source: {
loggerName: "stuff",
message: "stuff",
level: "stuff",
timestamp: "stuff",
thread: "stuff"
}
},

So, now I'm trying to actually query / sort this data. I'm building a
JSON request that looks like:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level" : { "order" : "asc" }
],
"query" : {
"term" : { "level" : targetLevel }
}
}

And execute the request with:

$.getJSON(myUrl, q, function(rData) {
console.log(rData);
});

However, what I get back is an unordered list of every log item,
rather than the sorted list that I'm looking for.

I've also tried:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level"
],
"query" : {
"term" : { "level" : targetLevel }
}
}

and a number of other variations, but haven't had much success.
Neither the search nor the sort seem to work.

I'm sure I must be making an obvious mistake. That said, if anyone
could point me in the right direction here, it would be greatly
appreciated :slight_smile:

Also, related question: is there a simple way to ask Elasticsearch to
be verbose about how / why it's filtering and sorting its results?

Thanks all!

--

--

--

5

David / Raffaele:

Explain is useful. Thanks!

Also, thanks for the discussion so far. I've simplified my JSON to:

q = {
"query" : {
"term" : { "level" : "INFO" }
},
"explain" : true
}

$.getJSON(sSource, q, function(rData) {
console.log(rData);
});

It looks like the query ES is running is effectively ':', because here's
a snippet of the "explain" I get back from the above query:

{

  • value: 1,
  • description: "ConstantScore(NotDeleted(:)), product of:",

  • details:
    [

    {
    - value: 1,
    - description: "boost"
    },

    {
    - value: 1,
    - description: "queryNorm"
    }
    ]

}

Any further thoughts?

On Thursday, October 4, 2012 12:10:44 PM UTC-4, David Pilato wrote:

Explain will give you clues on how score is computed. IMHO, you won't get
details why some results are filtered or not.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2012 à 18:07, Raffaele Sena <raf...@gmail.com <javascript:>> a
écrit :

for the second part of your question, just add "explain":true to your
json object. You'll get more information than you asked for, but it
may give you some clues.

On Thu, Oct 4, 2012 at 8:52 AM, Steve <cubi...@gmail.com <javascript:>>
wrote:

Hi all:

Just getting started with Elasticsearch. I've taken some log4j output
and inserted it into Elasticsearch to give me some data to play with,
which looks like the following:

{
total: 34,
max_score: 1,
hits: [
{
_index: "tomcat",
_type: "base",
_id: "an_id",
_score: 1,
_source: {
loggerName: "stuff",
message: "stuff",
level: "stuff",
timestamp: "stuff",
thread: "stuff"
}
},

So, now I'm trying to actually query / sort this data. I'm building a
JSON request that looks like:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level" : { "order" : "asc" }
],
"query" : {
"term" : { "level" : targetLevel }
}
}

And execute the request with:

$.getJSON(myUrl, q, function(rData) {
console.log(rData);
});

However, what I get back is an unordered list of every log item,
rather than the sorted list that I'm looking for.

I've also tried:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level"
],
"query" : {
"term" : { "level" : targetLevel }
}
}

and a number of other variations, but haven't had much success.
Neither the search nor the sort seem to work.

I'm sure I must be making an obvious mistake. That said, if anyone
could point me in the right direction here, it would be greatly
appreciated :slight_smile:

Also, related question: is there a simple way to ask Elasticsearch to
be verbose about how / why it's filtering and sorting its results?

Thanks all!

--

--

--

6

Ah, I think I see.

I'm making a GET request to _search, but I'm not correctly putting the
query DSL into the body of the request. Thus, my query is being attached
as a GET parameter instead of included as part of a request body.
Because ES doesn't see any parameters it understands, it gives back the
default ':', and I end up with something that isn't what I thought I
asked for.

Guess I need to go back and read more of the manual :slight_smile:

Thanks all!

On Thursday, October 4, 2012 1:42:55 PM UTC-4, cubic1271 wrote:

David / Raffaele:

Explain is useful. Thanks!

Also, thanks for the discussion so far. I've simplified my JSON to:

q = {
"query" : {
"term" : { "level" : "INFO" }
},
"explain" : true
}

$.getJSON(sSource, q, function(rData) {
console.log(rData);
});

It looks like the query ES is running is effectively ':', because here's
a snippet of the "explain" I get back from the above query:

{

  • value: 1,
  • description: "ConstantScore(NotDeleted(:)), product of:",

  • details:
    [

    {
    - value: 1,
    - description: "boost"
    },

    {
    - value: 1,
    - description: "queryNorm"
    }
    ]

}

Any further thoughts?

On Thursday, October 4, 2012 12:10:44 PM UTC-4, David Pilato wrote:

Explain will give you clues on how score is computed. IMHO, you won't get
details why some results are filtered or not.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2012 à 18:07, Raffaele Sena raf...@gmail.com a écrit :

for the second part of your question, just add "explain":true to your
json object. You'll get more information than you asked for, but it
may give you some clues.

On Thu, Oct 4, 2012 at 8:52 AM, Steve cubi...@gmail.com wrote:

Hi all:

Just getting started with Elasticsearch. I've taken some log4j output
and inserted it into Elasticsearch to give me some data to play with,
which looks like the following:

{
total: 34,
max_score: 1,
hits: [
{
_index: "tomcat",
_type: "base",
_id: "an_id",
_score: 1,
_source: {
loggerName: "stuff",
message: "stuff",
level: "stuff",
timestamp: "stuff",
thread: "stuff"
}
},

So, now I'm trying to actually query / sort this data. I'm building a
JSON request that looks like:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level" : { "order" : "asc" }
],
"query" : {
"term" : { "level" : targetLevel }
}
}

And execute the request with:

$.getJSON(myUrl, q, function(rData) {
console.log(rData);
});

However, what I get back is an unordered list of every log item,
rather than the sorted list that I'm looking for.

I've also tried:

q = {
"from": offset,
"size": numItems,
"sort" : [
"level"
],
"query" : {
"term" : { "level" : targetLevel }
}
}

and a number of other variations, but haven't had much success.
Neither the search nor the sort seem to work.

I'm sure I must be making an obvious mistake. That said, if anyone
could point me in the right direction here, it would be greatly
appreciated :slight_smile:

Also, related question: is there a simple way to ask Elasticsearch to
be verbose about how / why it's filtering and sorting its results?

Thanks all!

--

--

--