Any way to store JSON data without analyzing it


(Matt Schoen) #1

Hi there,

I started a topic earlier about making a mapping for a recursive tree
structure to which there wasn't a great answer, but my original desire was
to just turn off analysis on certain indexes. All that I'd like to do is
store a JSON document by ID and recall it. In this case the need has again
arisen because of the way I'm treating floating point numbers in my JSON
parser. I designed it to support INFINITY and NaN, which I understand is
not actually a part of the official JSON specification. The issue I have
is that sometimes a float field is a number, and sometimes it's a string.
For example, I have a field that can sometimes be = INFINITY. If I've
mapped this field as a float, a document with it = INFINITY will come in
with that field as a string. This throws an error:

MapperParsingException[failed to parse [max_length]]; nested:
NumberFormatException[For input string: "INFINITY"];

max_length being the field name and INFINITY being the value I'm trying to
set it to. As I said before I really just want to disable analysis and
store the documents wholesale. Or maybe I can just mark certain fields as
don't analyze? Any other ideas for what I should do here? The only thing
I can think of is to escape the JSON and store it as a string, or to
convert all my float parameters to and from strings when I store them.

Thanks for the help!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

you could achieve this, using mapping templates or simply ignore this kind
of errors when indexing. There is a ignore_malformed configuration setting
for the mapping.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html

--Alex

On Wed, Nov 6, 2013 at 11:17 AM, Matt Schoen mtschoen@gmail.com wrote:

Hi there,

I started a topic earlier about making a mapping for a recursive tree
structure to which there wasn't a great answer, but my original desire was
to just turn off analysis on certain indexes. All that I'd like to do is
store a JSON document by ID and recall it. In this case the need has again
arisen because of the way I'm treating floating point numbers in my JSON
parser. I designed it to support INFINITY and NaN, which I understand is
not actually a part of the official JSON specification. The issue I have
is that sometimes a float field is a number, and sometimes it's a string.
For example, I have a field that can sometimes be = INFINITY. If I've
mapped this field as a float, a document with it = INFINITY will come in
with that field as a string. This throws an error:

MapperParsingException[failed to parse [max_length]]; nested:
NumberFormatException[For input string: "INFINITY"];

max_length being the field name and INFINITY being the value I'm trying to
set it to. As I said before I really just want to disable analysis and
store the documents wholesale. Or maybe I can just mark certain fields as
don't analyze? Any other ideas for what I should do here? The only thing
I can think of is to escape the JSON and store it as a string, or to
convert all my float parameters to and from strings when I store them.

Thanks for the help!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Matt Schoen) #3

Hey Alex. Your advice led me down the right path. The enabled property was
ultimately my friend.

My mapping now has
{ ...

"properties"{

"data":{"type":"object", "enabled":false}

}

}

and everything is happy! Hallelujah! It would be nice for this to be a
little clearer in the documentation. It seems that this would be a very
common need and the enabled flag is kind of buried. Enabled is mentioned
in the first paragraph but its function isn't clear until later. Anyway
now I know so I'm all set, but I wonder if we could make this process
easier for noobs.

On Thursday, November 14, 2013 3:07:09 AM UTC-7, Alexander Reelsen wrote:

Hey,

you could achieve this, using mapping templates or simply ignore this kind
of errors when indexing. There is a ignore_malformed configuration setting
for the mapping.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html

--Alex

On Wed, Nov 6, 2013 at 11:17 AM, Matt Schoen <mtsc...@gmail.com<javascript:>

wrote:

Hi there,

I started a topic earlier about making a mapping for a recursive tree
structure to which there wasn't a great answer, but my original desire was
to just turn off analysis on certain indexes. All that I'd like to do is
store a JSON document by ID and recall it. In this case the need has again
arisen because of the way I'm treating floating point numbers in my JSON
parser. I designed it to support INFINITY and NaN, which I understand is
not actually a part of the official JSON specification. The issue I have
is that sometimes a float field is a number, and sometimes it's a string.
For example, I have a field that can sometimes be = INFINITY. If I've
mapped this field as a float, a document with it = INFINITY will come in
with that field as a string. This throws an error:

MapperParsingException[failed to parse [max_length]]; nested:
NumberFormatException[For input string: "INFINITY"];

max_length being the field name and INFINITY being the value I'm trying
to set it to. As I said before I really just want to disable analysis and
store the documents wholesale. Or maybe I can just mark certain fields as
don't analyze? Any other ideas for what I should do here? The only thing
I can think of is to escape the JSON and store it as a string, or to
convert all my float parameters to and from strings when I store them.

Thanks for the help!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #4