Problems with dynamic mapping

HI folks,

We are using elastic search index a bunch of JSON documents which all have *
roughly* the same structure, but there are some variations which is causing
headaches with dynamic mapping. Specifically fields in the document that,
when first observed by elastic search, were numeric types (and therefore
mapped as numeric by ES), but in later documents these same fields have
string values. This results in a NumberFormaException in ES and the
document cannot be indexed.

A couple of caveats:

  • We don't control the documents. They come from an external source and
    are "supposed" to conform to a standard, but often don't
  • We can't plausibly provide a mapping for the entire document. They
    are huge (in structure) and not always consistent.

We do provide a custom mapping for the fields we really care about
filtering on, but the others we we want indexed but care a little less
about how we would filter on these values (if at all).

Is there a way to tell elastic search to treat any new field it sees as a
string? I.e., don't infer the type for a new field, just treat it as a
string?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You can use dynamic templates to control how dynamic fields (the ones not
explicitly defined in your mapping) are mapped:

Elasticsearch Platform — Find real-time answers at scale | Elastic(scroll
down for dynamic templates)

--
Ivan

On Wed, Mar 6, 2013 at 11:41 AM, Jason jason.polites@getsocialize.comwrote:

HI folks,

We are using Elasticsearch index a bunch of JSON documents which all have
roughly the same structure, but there are some variations which is
causing headaches with dynamic mapping. Specifically fields in the
document that, when first observed by Elasticsearch, were numeric types
(and therefore mapped as numeric by ES), but in later documents these same
fields have string values. This results in a NumberFormaException in ES
and the document cannot be indexed.

A couple of caveats:

  • We don't control the documents. They come from an external source
    and are "supposed" to conform to a standard, but often don't
  • We can't plausibly provide a mapping for the entire document. They
    are huge (in structure) and not always consistent.

We do provide a custom mapping for the fields we really care about
filtering on, but the others we we want indexed but care a little less
about how we would filter on these values (if at all).

Is there a way to tell Elasticsearch to treat any new field it sees as a
string? I.e., don't infer the type for a new field, just treat it as a
string?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Perfect. Exactly what I needed. Thanks! (not sure how I missed this)

On Wed, Mar 6, 2013 at 2:22 PM, Ivan Brusic ivan@brusic.com wrote:

You can use dynamic templates to control how dynamic fields (the ones not
explicitly defined in your mapping) are mapped:

Elasticsearch Platform — Find real-time answers at scale | Elastic(scroll down for dynamic templates)

--
Ivan

On Wed, Mar 6, 2013 at 11:41 AM, Jason jason.polites@getsocialize.comwrote:

HI folks,

We are using Elasticsearch index a bunch of JSON documents which all
have roughly the same structure, but there are some variations which
is causing headaches with dynamic mapping. Specifically fields in the
document that, when first observed by Elasticsearch, were numeric types
(and therefore mapped as numeric by ES), but in later documents these same
fields have string values. This results in a NumberFormaException in ES
and the document cannot be indexed.

A couple of caveats:

  • We don't control the documents. They come from an external source
    and are "supposed" to conform to a standard, but often don't
  • We can't plausibly provide a mapping for the entire document. They
    are huge (in structure) and not always consistent.

We do provide a custom mapping for the fields we really care about
filtering on, but the others we we want indexed but care a little less
about how we would filter on these values (if at all).

Is there a way to tell Elasticsearch to treat any new field it sees as a
string? I.e., don't infer the type for a new field, just treat it as a
string?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/gIcNdmIXo_Y/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Jason Polites*
*
Socialize Inc*
*
Jason.Polites@GetSocialize.com jason.polites@getsocialize.com*
*153 Townsend St #9057
San Francisco
CA 94107

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.