Search doesn't find all documents

blake1024 · April 13, 2015, 7:02pm

Greetings. I am using elasticsearch 1.5.1 on a 64 bit Linux box. I am
placing a large number of documents in a single index, but when I search,
it only finds a few of the matching documents - not all. Using other
searches, I can validate that a particular document is present, but when I
do the search I am trying to get working, it doesn't find the document.
For example:

Using the following search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{
"match" : {
"classifications" : "passive_components/resistors/single_components"
}
}
]
}
}
}
'

I get the following document, among many others:

  {
    _index: "components-1428943055325",
    _type: "component",
    _id: "4ccf8ca7-ed99-46a8-8a43-9636099ada8f",
    _score: 1.0630159,
    _source: {
      name: "R 1206 50.0m 0.25W 1% Thick Film",
      classifications: "passive_components/resistors/single_components",
      prominentProperties: [
        {
          name: "Resistance",
          id: "property.id-5b532445-5032-4c0a-86c3-d07a69a80ef8",
          value: "0.05",
          units: "ohm"
        }
      ],
      otherProperties: [
        {
          name: "Cp",
          id: "property.id-c36a6fff-dbce-4757-98af-d065466f33ba",
          value: "0.07e-12",
          units: "F"
        },
        {
          name: "Octopart mpn",
          id: "property.id-e8af007b-ef99-43bb-b074-610dcc9b8762",
          value: "RL1206FR-070R05L"
        },
        {
          name: "Ls",
          id: "property.id-520d1642-dd28-4634-8f6d-5c927e58b16d",
          value: "0.95e-9",
          units: "H"
        }
      ],
      position: 6099
    }
  }

However, with this search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{and
"match" : {
"value" : "0.05"
}
}
]
}
}
}
'

I get back only a few documents, but the above document is not in there.
I tried 0.05 without the quotes with no help. Also, the reason I am using
"bool" and "must" is because this is actually a simplified version of the
real query.

Sure appreciate any help!

Blake McBride

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

blake1024 · April 13, 2015, 7:22pm

The spurious "and" in the last query is a copy/paste error and not in the
query I am using.

On Monday, April 13, 2015 at 2:02:43 PM UTC-5, Blake McBride wrote:

Greetings. I am using elasticsearch 1.5.1 on a 64 bit Linux box. I am
placing a large number of documents in a single index, but when I search,
it only finds a few of the matching documents - not all. Using other
searches, I can validate that a particular document is present, but when I
do the search I am trying to get working, it doesn't find the document.
For example:

Using the following search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{
"match" : {
"classifications" : "passive_components/resistors/single_components"
}
}
]
}
}
}
'

I get the following document, among many others:
  {
    _index: "components-1428943055325",
    _type: "component",
    _id: "4ccf8ca7-ed99-46a8-8a43-9636099ada8f",
    _score: 1.0630159,
    _source: {
      name: "R 1206 50.0m 0.25W 1% Thick Film",
      classifications: 
"passive_components/resistors/single_components",
prominentProperties: [
{
name: "Resistance",
id: "property.id-5b532445-5032-4c0a-86c3-d07a69a80ef8",
value: "0.05",
units: "ohm"
}
],
otherProperties: [
{
name: "Cp",
id: "property.id-c36a6fff-dbce-4757-98af-d065466f33ba",
value: "0.07e-12",
units: "F"
},
{
name: "Octopart mpn",
id: "property.id-e8af007b-ef99-43bb-b074-610dcc9b8762",
value: "RL1206FR-070R05L"
},
{
name: "Ls",
id: "property.id-520d1642-dd28-4634-8f6d-5c927e58b16d",
value: "0.95e-9",
units: "H"
}
],
position: 6099
}
}

However, with this search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{and
"match" : {
"value" : "0.05"
}
}
]
}
}
}
'

I get back only a few documents, but the above document is not in
there. I tried 0.05 without the quotes with no help. Also, the reason I
am using "bool" and "must" is because this is actually a simplified version
of the real query.

Sure appreciate any help!

Blake McBride

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8a732066-50e4-4b23-8368-55ccdd1268e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

lesbarstow · April 13, 2015, 8:22pm

I would start with this line of thought: have you explicitly set the field
"value" to be a numeric type like "float" in your index template?

--
Les Barstow, Senior Software Engineer
Return Path, Inc.

On Mon, Apr 13, 2015 at 1:02 PM, Blake McBride blake1024@gmail.com wrote:

Greetings. I am using elasticsearch 1.5.1 on a 64 bit Linux box. I am
placing a large number of documents in a single index, but when I search,
it only finds a few of the matching documents - not all. Using other
searches, I can validate that a particular document is present, but when I
do the search I am trying to get working, it doesn't find the document.
For example:

Using the following search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{
"match" : {
"classifications" : "passive_components/resistors/single_components"
}
}
]
}
}
}
'

I get the following document, among many others:
  {
    _index: "components-1428943055325",
    _type: "component",
    _id: "4ccf8ca7-ed99-46a8-8a43-9636099ada8f",
    _score: 1.0630159,
    _source: {
      name: "R 1206 50.0m 0.25W 1% Thick Film",
      classifications:
"passive_components/resistors/single_components",
prominentProperties: [
{
name: "Resistance",
id: "property.id-5b532445-5032-4c0a-86c3-d07a69a80ef8",
value: "0.05",
units: "ohm"
}
],
otherProperties: [
{
name: "Cp",
id: "property.id-c36a6fff-dbce-4757-98af-d065466f33ba",
value: "0.07e-12",
units: "F"
},
{
name: "Octopart mpn",
id: "property.id-e8af007b-ef99-43bb-b074-610dcc9b8762",
value: "RL1206FR-070R05L"
},
{
name: "Ls",
id: "property.id-520d1642-dd28-4634-8f6d-5c927e58b16d",
value: "0.95e-9",
units: "H"
}
],
position: 6099
}
}

However, with this search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{and
"match" : {
"value" : "0.05"
}
}
]
}
}
}
'

I get back only a few documents, but the above document is not in
there. I tried 0.05 without the quotes with no help. Also, the reason I
am using "bool" and "must" is because this is actually a simplified version
of the real query.

Sure appreciate any help!

Blake McBride

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOppbCXy%3DJqHMSDRigdDrjkU9UDSwegQQVcmu4y7dkmj%2BDNxPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

softwaredoug · April 13, 2015, 8:31pm

Well it looks like classifications is a property of the root object,
"value" is an inner object, so you'd need to query for it accordingly

Have you tried querying *"**prominentProperties.value" *in your searches?

You might want to read on data modeling

Cheers,

*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections,
LLC | 240.476.9983 | http://www.opensourceconnections.com
Author: Taming Search http://manning.com/turnbull from Manning
Publications
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

On Mon, Apr 13, 2015 at 3:02 PM, Blake McBride blake1024@gmail.com wrote:

Greetings. I am using elasticsearch 1.5.1 on a 64 bit Linux box. I am
placing a large number of documents in a single index, but when I search,
it only finds a few of the matching documents - not all. Using other
searches, I can validate that a particular document is present, but when I
do the search I am trying to get working, it doesn't find the document.
For example:

Using the following search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{
"match" : {
"classifications" : "passive_components/resistors/single_components"
}
}
]
}
}
}
'

I get the following document, among many others:
  {
    _index: "components-1428943055325",
    _type: "component",
    _id: "4ccf8ca7-ed99-46a8-8a43-9636099ada8f",
    _score: 1.0630159,
    _source: {
      name: "R 1206 50.0m 0.25W 1% Thick Film",
      classifications:
"passive_components/resistors/single_components",
prominentProperties: [
{
name: "Resistance",
id: "property.id-5b532445-5032-4c0a-86c3-d07a69a80ef8",
value: "0.05",
units: "ohm"
}
],
otherProperties: [
{
name: "Cp",
id: "property.id-c36a6fff-dbce-4757-98af-d065466f33ba",
value: "0.07e-12",
units: "F"
},
{
name: "Octopart mpn",
id: "property.id-e8af007b-ef99-43bb-b074-610dcc9b8762",
value: "RL1206FR-070R05L"
},
{
name: "Ls",
id: "property.id-520d1642-dd28-4634-8f6d-5c927e58b16d",
value: "0.95e-9",
units: "H"
}
],
position: 6099
}
}

However, with this search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{and
"match" : {
"value" : "0.05"
}
}
]
}
}
}
'

I get back only a few documents, but the above document is not in
there. I tried 0.05 without the quotes with no help. Also, the reason I
am using "bool" and "must" is because this is actually a simplified version
of the real query.

Sure appreciate any help!

Blake McBride

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALG6HL_uOii%3Dso-A%2B1CLVF45%3DWVVtK_0hvD88RuSNpMc6Q4JmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

blake1024 · April 13, 2015, 10:00pm

Interestingly, when I let ES build the schema, it doesn't even list those
elements. I thought that might be because they were too deep or because
they are in an array. Not knowing how ES is dealing with the value field,
I also tried dealing with it like a number giving a rang of (> 0.04 AND <
0.06) but no help.

On Monday, April 13, 2015 at 3:22:28 PM UTC-5, Les Barstow wrote:

I would start with this line of thought: have you explicitly set the field
"value" to be a numeric type like "float" in your index template?

--
Les Barstow, Senior Software Engineer
Return Path, Inc.

On Mon, Apr 13, 2015 at 1:02 PM, Blake McBride <blak...@gmail.com
<javascript:>> wrote:
Greetings. I am using elasticsearch 1.5.1 on a 64 bit Linux box. I am
placing a large number of documents in a single index, but when I search,
it only finds a few of the matching documents - not all. Using other
searches, I can validate that a particular document is present, but when I
do the search I am trying to get working, it doesn't find the document.
For example:

Using the following search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{
"match" : {
"classifications" : "passive_components/resistors/single_components"
}
}
]
}
}
}
'

I get the following document, among many others:
  {
    _index: "components-1428943055325",
    _type: "component",
    _id: "4ccf8ca7-ed99-46a8-8a43-9636099ada8f",
    _score: 1.0630159,
    _source: {
      name: "R 1206 50.0m 0.25W 1% Thick Film",
      classifications: 
"passive_components/resistors/single_components",
prominentProperties: [
{
name: "Resistance",
id: "property.id-5b532445-5032-4c0a-86c3-d07a69a80ef8",
value: "0.05",
units: "ohm"
}
],
otherProperties: [
{
name: "Cp",
id: "property.id-c36a6fff-dbce-4757-98af-d065466f33ba",
value: "0.07e-12",
units: "F"
},
{
name: "Octopart mpn",
id: "property.id-e8af007b-ef99-43bb-b074-610dcc9b8762",
value: "RL1206FR-070R05L"
},
{
name: "Ls",
id: "property.id-520d1642-dd28-4634-8f6d-5c927e58b16d",
value: "0.95e-9",
units: "H"
}
],
position: 6099
}
}

However, with this search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{and
"match" : {
"value" : "0.05"
}
}
]
}
}
}
'

I get back only a few documents, but the above document is not in
there. I tried 0.05 without the quotes with no help. Also, the reason I
am using "bool" and "must" is because this is actually a simplified version
of the real query.

Sure appreciate any help!

Blake McBride

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eaa4fbe4-fb1f-4d90-b566-7c871e4d355c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

blake1024 · April 13, 2015, 10:19pm

Yes. I think the data modeling pointer you gave me will help a lot. I
will go down that path. Thanks!

On Monday, April 13, 2015 at 3:31:45 PM UTC-5, Doug Turnbull wrote:

Well it looks like classifications is a property of the root object,
"value" is an inner object, so you'd need to query for it accordingly

Have you tried querying *"**prominentProperties.value" *in your searches?

You might want to read on data modeling

Modeling Your Data | Elasticsearch: The Definitive Guide [master] | Elastic

Cheers,

*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections,
LLC | 240.476.9983 | http://www.opensourceconnections.com
Author: Taming Search http://manning.com/turnbull from Manning
Publications
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

On Mon, Apr 13, 2015 at 3:02 PM, Blake McBride <blak...@gmail.com
<javascript:>> wrote:
Greetings. I am using elasticsearch 1.5.1 on a 64 bit Linux box. I am
placing a large number of documents in a single index, but when I search,
it only finds a few of the matching documents - not all. Using other
searches, I can validate that a particular document is present, but when I
do the search I am trying to get working, it doesn't find the document.
For example:

Using the following search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{
"match" : {
"classifications" : "passive_components/resistors/single_components"
}
}
]
}
}
}
'

I get the following document, among many others:
  {
    _index: "components-1428943055325",
    _type: "component",
    _id: "4ccf8ca7-ed99-46a8-8a43-9636099ada8f",
    _score: 1.0630159,
    _source: {
      name: "R 1206 50.0m 0.25W 1% Thick Film",
      classifications: 
"passive_components/resistors/single_components",
prominentProperties: [
{
name: "Resistance",
id: "property.id-5b532445-5032-4c0a-86c3-d07a69a80ef8",
value: "0.05",
units: "ohm"
}
],
otherProperties: [
{
name: "Cp",
id: "property.id-c36a6fff-dbce-4757-98af-d065466f33ba",
value: "0.07e-12",
units: "F"
},
{
name: "Octopart mpn",
id: "property.id-e8af007b-ef99-43bb-b074-610dcc9b8762",
value: "RL1206FR-070R05L"
},
{
name: "Ls",
id: "property.id-520d1642-dd28-4634-8f6d-5c927e58b16d",
value: "0.95e-9",
units: "H"
}
],
position: 6099
}
}

However, with this search:

curl -s -XGET 'http://localhost:9200/components/_search' -d '{
"from" : 0,
"size" : 100000,
"query" : {
"bool" : {
"must" : [
{and
"match" : {
"value" : "0.05"
}
}
]
}
}
}
'

I get back only a few documents, but the above document is not in
there. I tried 0.05 without the quotes with no help. Also, the reason I
am using "bool" and "must" is because this is actually a simplified version
of the real query.

Sure appreciate any help!

Blake McBride

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/779fef4f-619b-4f13-bc4e-145e06163ed1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a429547c-e705-471d-888b-870ada31dc72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.