Cannot search in sub-attribute

Hello,

I created a doc type which has several attributes including an attribute
@fields configured like this:

"@fields": {
"type": "object",
"index": "not_analyzed",
"dynamic": true,
"path": "full"
}

Here is a sample input:

{
"@fields": {
"hostname": "walter",
"ip": "192.168.10.237",
"user": {
"firstname": "Donny",
"lastname": "Kerabatsos"
},
"message": "Shut the f* up, Donny"
}
}

Now I can search for a doc by referencing the hostname like this:

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.hostname": "walter"
}
}
]
}
}
}
}

But this search yields* no result:*

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.user.firstname": "Donny"
}
}
]
}
}
}
}

I don't understand why, but perhaps I misunderstood something about the
object type. My impression was, that it is possible to insert an arbitrary
json and perform searches on any of it's attributes (also "nested"
attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json
data which is inserted into @fields changes (different hierarchies,
attribute names, etc.).

Thanks a lot & cheers,

  • Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

you should use the match query instead of the term query, since the internal fields ends up being analyzed.

I see you try to set the index field on the object mapping to mark it as not_analyzed, this will not work and is not supported (I think you assume that al internal mapping will inherit it, which they do not). if you want to achieve this, check into dynamic templates in the root object mappings.

On Sat, Jun 1, 2013 at 4:18 PM, Chris Kinsalb hc.kinsalb@gmail.com
wrote:

Hello,
I created a doc type which has several attributes including an attribute
@fields configured like this:
"@fields": {
"type": "object",
"index": "not_analyzed",
"dynamic": true,
"path": "full"
}
Here is a sample input:
{
"@fields": {
"hostname": "walter",
"ip": "192.168.10.237",
"user": {
"firstname": "Donny",
"lastname": "Kerabatsos"
},
"message": "Shut the f* up, Donny"
}
}
Now I can search for a doc by referencing the hostname like this:
{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.hostname": "walter"
}
}
]
}
}
}
}
But this search yields* no result:*
{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.user.firstname": "Donny"
}
}
]
}
}
}
}
I don't understand why, but perhaps I misunderstood something about the
object type. My impression was, that it is possible to insert an arbitrary
json and perform searches on any of it's attributes (also "nested"
attributes), due to the dynamic mapping.
1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json
data which is inserted into @fields changes (different hierarchies,
attribute names, etc.).
Thanks a lot & cheers,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello Shay,

thanks for the explanation.
Yes, that is indeed what I thought. However, the problem is that I wanted
my input to be totally schema-less, which I cannot seem to achieve using
dynamic templates since I will not know what the input objects will look
like.

However, I do have an idea on how to achieve this:

Using dynamic templates could I do something along these lines?

    "dynamic_templates": [
        {
          "fields_template": {
            "mapping": {
              "index": "not_analyzed"
            },
            "match": "fields.*"
          }
        }
      ]

Now, whenever I get a json input I would transform it to a flat hierarchy,
e.g.:

input:

{
foo: 'bar',
user: {
firstname: 'bla',
lastname: 'narf'
}
}

Transform output:

{
fields.foo: 'bar',
fields.user.firstname: 'bla',
fields.user.lastname: 'narf'
}

This way:

  • each field will automatically have the correct type (e.g. string or
    integer)
  • I can run facet queries (statistics) on the fields

Does this sound like a viable solution to you?
(will try it out now)

Cheers & regards,

  • Chris

On Saturday, June 1, 2013 5:19:40 PM UTC+2, kimchy wrote:

you should use the match query instead of the term query, since the
internal fields ends up being analyzed.

I see you try to set the index field on the object mapping to mark it as
not_analyzed, this will not work and is not supported (I think you assume
that al internal mapping will inherit it, which they do not). if you want
to achieve this, check into dynamic templates in the root object mappings.

On Sat, Jun 1, 2013 at 4:18 PM, Chris Kinsalb <hc.ki...@gmail.com<javascript:>

wrote:

Hello,

I created a doc type which has several attributes including an attribute
@fields configured like this:

"@fields": {
"type": "object",
"index": "not_analyzed",
"dynamic": true,
"path": "full"
}

Here is a sample input:

{
"@fields": {
"hostname": "walter",
"ip": "192.168.10.237",
"user": {
"firstname": "Donny",
"lastname": "Kerabatsos"
},
"message": "Shut the f* up, Donny"
}
}

Now I can search for a doc by referencing the hostname like this:

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.hostname": "walter"
}
}
]
}
}
}
}

But this search yields* no result:*

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.user.firstname": "Donny"
}
}
]
}
}
}
}

I don't understand why, but perhaps I misunderstood something about the
object type. My impression was, that it is possible to insert an arbitrary
json and perform searches on any of it's attributes (also "nested"
attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json
data which is inserted into @fields changes (different hierarchies,
attribute names, etc.).

Thanks a lot & cheers,

  • Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

fyi:

My own suggested solution works as intended. I can now add arbitrarily
nested objects, with each attribute flattened down to its full path.

The query to server:9200/index/{doctype}/ _mapping shows that the field
types are created correctly was well.

  • chris

On Monday, June 3, 2013 8:18:54 AM UTC+2, Chris Kinsalb wrote:

Hello Shay,

thanks for the explanation.
Yes, that is indeed what I thought. However, the problem is that I wanted
my input to be totally schema-less, which I cannot seem to achieve using
dynamic templates since I will not know what the input objects will look
like.

However, I do have an idea on how to achieve this:

Using dynamic templates could I do something along these lines?

    "dynamic_templates": [
        {
          "fields_template": {
            "mapping": {
              "index": "not_analyzed"
            },
            "match": "fields.*"
          }
        }
      ]

Now, whenever I get a json input I would transform it to a flat hierarchy,
e.g.:

input:

{
foo: 'bar',
user: {
firstname: 'bla',
lastname: 'narf'
}
}

Transform output:

{
fields.foo: 'bar',
fields.user.firstname: 'bla',
fields.user.lastname: 'narf'
}

This way:

  • each field will automatically have the correct type (e.g. string or
    integer)
  • I can run facet queries (statistics) on the fields

Does this sound like a viable solution to you?
(will try it out now)

Cheers & regards,

  • Chris

On Saturday, June 1, 2013 5:19:40 PM UTC+2, kimchy wrote:

you should use the match query instead of the term query, since the
internal fields ends up being analyzed.

I see you try to set the index field on the object mapping to mark it as
not_analyzed, this will not work and is not supported (I think you assume
that al internal mapping will inherit it, which they do not). if you want
to achieve this, check into dynamic templates in the root object mappings.

On Sat, Jun 1, 2013 at 4:18 PM, Chris Kinsalb hc.ki...@gmail.com wrote:

Hello,

I created a doc type which has several attributes including an
attribute @fields configured like this:

"@fields": {
"type": "object",
"index": "not_analyzed",
"dynamic": true,
"path": "full"
}

Here is a sample input:

{
"@fields": {
"hostname": "walter",
"ip": "192.168.10.237",
"user": {
"firstname": "Donny",
"lastname": "Kerabatsos"
},
"message": "Shut the f* up, Donny"
}
}

Now I can search for a doc by referencing the hostname like this:

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.hostname": "walter"
}
}
]
}
}
}
}

But this search yields* no result:*

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"@fields.user.firstname": "Donny"
}
}
]
}
}
}
}

I don't understand why, but perhaps I misunderstood something about the
object type. My impression was, that it is possible to insert an arbitrary
json and perform searches on any of it's attributes (also "nested"
attributes), due to the dynamic mapping.

1.) What do I have to do in order to make this work?
2.) I DO not want to create a default schema for this, since the json
data which is inserted into @fields changes (different hierarchies,
attribute names, etc.).

Thanks a lot & cheers,

  • Chris

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.