Having trouble with MappingParserExceptions... any ideas?

Hey all. I've been trying to use ElasticSearch at work for 2 months now,
but keep running into the same problem. When indexing JSON objects, I get
errors like this:

{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected end-of-input within/between OBJECT entries\n
at [Source: [B@1dd6a67; line: 1, column: 36145]]; ","status":400}

Sometimes the text after "Unexpected end-of-input" is different, but it
doesn't matter, because the result is the same: the document is not
indexed. Now, to give you some background, I'm using a Node.js
ElasticSearch client to send these JSON objects, but I'm sure that is not
the problem, since I don't get these errors when using simple test objects
(2 months of debugging has led to my brazen certainty :~)). The documents
I'm trying to index from another production data store (MongoDB) are big,
complex JSON docs. I know that if the doc is different from the mapping,
i'll have issues, but there are certain fields in our docs that may be
either strings or arrays. So, I added index.mapping.ignore_malformed to my
elasticsearch.yml (I'm using the RC version 0.19.11 of ES, to take
advantage of that ignore_malformed feature). I thought that would take care
of things, but alas, it hasn't. So then I thought, Maybe I should encode
these docs in Base64 to avoid non-standard character problems that may be
happening, since this end-of-input thing could be caused by that sort of
issue. However, I notice that on the elasticsearch.org documentation site,
Base64 encoding is only mentioned when talking about the attachments plugin
(allowing for .pdf, .doc, etc.). Since I'm just indexing JSON though, that
doesn't seem to be answer, but I'm uncertain. Is there any benefit to
encoding JSON in Base64 for ES, and if so, should I just do that then index
it normally, without using the attachment plugin? I'm just looking for some
guidance. I apologize if this post is all over the place, but I'm at my
wit's end with a product that I was very excited about getting into
production, but have been unable to.

An example of one of our documents:

{ _id: '12345',
item_id: '54321',
kind: 'article',
author: 'Inkoo Kang',
categories: [ 'Review', 'frankenweenie', 'review', 'tim burton' ],
content: '

frankenweenie-review

It’s
too bad that Frankenweenie
comes at this late-middle stage of Tim
Burton
’s career when the director, now more brand than auteur, has lost
his older fans, because it’s exactly the kind of funny, creepy, poignant
film many of us have been waiting for Burton to make since Edward
Scissorhands
and The Nightmare Before Christmas. It’s
probably too early to peg Frankenweenie as Burton’s comeback
vehicle, but it’s certainly the director’s best movie in twenty
years.\n\nAs with his best films, the origins of Frankenweenie
hail from Burton’s early years. The new movie expands a 30-minute short the
young director made for Disney, who famously fired him for making such a
morbid spectacle for children. Fans of the short, now a cult classic, can
breathe a sigh of relief; Burton has expanded the story cleverly and
meaningfully, and the feature feels less like a remake than the
self-actualization of a great idea.

\n

Frankenweenie takes
place in the generic, timeless suburb of New Holland, notable only for its
freakishly frequent thunderstorms. When the school’s science teacher gets
struck by lightning, Mr. Rzykruski (Martin Landau), an Old World soul with
Vincent Price’s face, comes to substitute. He quickly becomes a source of
inspiration to Victor (Charlie Tahan), a tall, lanky, slightly sullen boy
whose only friend is his dog Sparky. (The 1984 short’s bull terrier is
replaced by a more normal-looking dog breed, probably to the benefit of
bull terrier puppies.)

\n

Alas, Sparky is dispatched in short order
via sudden automobile encounter, and Victor, because his last name is
Frankenstein, decides to bring his dog back to life the old-fashioned way:
by digging up its body, lugging it back to his attic lab, and striking it
with lightning. Victor has a hard enough time keeping Sparky’s resurrection
from his parents, but things get more complicated when his competitive
classmates find out about the undead dog, assume Victor revivified Sparky
for the science fair (a guaranteed first-place win!), and repeat the
experiment with their own long-dead pets with drastically more dangerous
results.

\n

Victor is a cute dud of a protagonist, but the bizarre
side characters around him are absolutely delightful. His sort-of bully is
the mouth-breathing Edgar “E.” Gore (Atticus Shaffer), an ambitious dimwit
who’s a perfect amalgamation of the hunchback sidekick of yore and the
weird kid in school who drowns his bologna sandwiches in chocolate milk
before eating it with a fork. A perpetually terrified schoolmate (Catharine
O’Hara) stalks Victor while cradling a Bond villain cat with one hand and
clutching its poop with the other, intoning that the fluffball’s produced
an omen Victor must heed. (The poop is the omen.) Sparky himself is a
refreshingly doglike dog, none too bright and rarely cloying.

\n

But
the standout character is Mr. Rzykruski, the new science teacher, who has a
face like a weathered tribal mask that’s somehow warm and friendly. He’s
also an unabashed elitist, and goes on a hilarious pro-science rant at a
PTA meeting that might not go over well in certain red corners of America,
but whose reason and passion can’t be argued
with.

\n

Frankenweenie doesn’t offer much in the way of new
visual motifs; if you’ve seen a Burton film once or twice, you won’t be
surprised by anything here. Still, the stop-motion animation is as
beautiful as the medium has to offer. Shot in atmospheric black-and-white,
the puppets come alive through variegated textures and unexpected angles.
(The use of 3-D theatrics is surprisingly spare, so the $3 surcharge
probably isn’t necessary to enjoy the film to its fullest.)

\n

Burton’s visual team at Disney has imagineered a carefully detailed world that weds Brancusi-like character design to recognizable laws of physics, creating a world that feels simultaneously familiar and strange. And for the first time in a long time, Burton’s obsessive love of creature features serves him well, with subtle, clever allusions to B-movies and recent thrillers. Best of all, the film’s morbidity is disgusting in a perversely joyous way, going beyond aestheticizing skeletons to mining for humor in the squirming, rotting beauty of life after life.

\n

The frank simplicity of Frankenweenie is certainly crucial to its success, especially compared to the director’s overstuffed, convoluted Alice and Dark Shadows. All the same, it’s hard to resist wanting Burton to try just a little harder, especially when it comes to his lazy spoofing of the suburbs. The tale of a lonely misfit trying to escape the stultifying blandness of his pastel hell is a story Burton’s told time and again — done best in Edward Scissorhands, but also in Beetlejuice, Alice in Wonderland, and Corpse Bride.

\n

The young Burton grew up in the ticky-tacky town of Burbank, California, and so it’s easy to understand why the director so often uses the ‘burbs as the straw man in his films. He wants his heroes to be mavericks, even progressives, polar opposites of characters like the traditional, cartoonishly patriarchal mayor in Frankenweenie who demands everyone be just like him.

\n

But in the post-Douglas Sirk, post-Desperate Housewives, post-gentrification era of today, Burton’s “social satire” seems not only dated, but irrelevant. And too often, Burton’s gentle excoriation of the rigid social values of the suburbs serves as a cover for the fact that he is one of the most socially conservative mainstream filmmakers working today. His casts and characters are almost shockingly whitewashed, even for Hollywood, and his oeuvre brims full with passive, doll-eyed heroines in need of rescue. (Even the dragon-slaying Alice, Burton’s only female protagonist in thirty years of filmmaking, needed plenty of saving.)

\n

In Frankenweenie’s New Holland, for example, the sole nonwhite resident is a vaguely villainous Japanese boy who fits all the negative stereotypes of Asian Americans: haughty, foreign, smart but uncreative. And when the students’ experiments go awry, Frankenweenie works up to a thrilling, satisfying climax, but the plot conspicuously twists and contorts to have its two male heroes take turns heroically saving their love interests. Tim Burton obviously revels in old-fashioned styles, but that’s no excuse for outdated sensibilities.

\n

Inkoo Kang is a Boston-based film journalist and regular contributor to BoxOffice Magazine whose work has appeared in Pop Matters and Screen Junkies. She reviews stuff she hates, likes, and hate-likes on her blog THINK-O-VISION.

\n

Follow Inkoo Kang on Twitter.\nFollow Movieline on Twitter.

', excerpt: 'It’s too bad that Frankenweenie comes at this late-middle stage of Tim Burton’s career when the director, now more brand than auteur, has lost his older fans, because it’s exactly the kind of funny, creepy, poignant film many of us have been waiting for Burton to make since Edward Scissorhands and The Nightmare Before Christmas. It’s probably too early to peg Frankenweenie as Burton’s comeback vehicle, but it’s certainly the director’s best movie in twenty years.\n\nAs with his best films, the origins of Frankenweenie hail from Burton’s early years. The new movie expands a 30-minute short the young director made for Disney, who famously fired him for making such a morbid spectacle for children. Fans of the short, now a cult classic, can breathe a sigh of relief; Burton has expanded the story cleverly and meaningfully, and the feature feels less like a remake than the self-actualization of a great idea. \nFrankenweenie takes place in the generic, timeless suburb of New Holland, notable only for its freakishly frequent thunderstorms. When the school’s science teacher gets struck by lightning, Mr. Rzykruski (Martin Landau), an Old World soul with Vincent Price’s face, comes to substitute. He quickly becomes a source of inspiration to Victor (Charlie Tahan), a tall, lanky, slightly sullen boy whose only friend is his dog Sparky. (The 1984 short’s bull terrier is replaced by a more normal-looking dog breed, probably to the benefit of bull terrier puppies.) \nAlas, Sparky is dispatched in short order via sudden automobile encounter, and Victor, because his last name is Frankenstein, decides to bring his dog back to life the old-fashioned way: by digging up its body, lugging it back to his attic lab, and striking it with lightning. Victor has a hard enough time keeping Sparky’s resurrection from his parents, but things get more complicated when his competitive classmates find out about the undead dog, assume Victor revivified Sparky for the science fair (a guaranteed first-place win!), and repeat the experiment with their own long-dead pets with drastically more dangerous results. \nVictor is a cute dud of a protagonist, but the bizarre side characters around him are absolutely delightful. His sort-of bully is the mouth-breathing Edgar “E.” Gore (Atticus Shaffer), an ambitious dimwit who’s a perfect amalgamation of the hunchback sidekick of yore and the weird kid in school who drowns his bologna sandwiches in chocolate milk before eating it with a fork. A perpetually terrified schoolmate (Catharine O’Hara) stalks Victor while cradling a Bond villain cat with one hand and clutching its poop with the other, intoning that the fluffball’s produced an omen Victor must heed. (The poop is the omen.) Sparky himself is a refreshingly doglike dog, none too bright and rarely cloying.\nBut the standout character is Mr. Rzykruski, the new science teacher, who has a face like a weathered tribal mask that’s somehow warm and friendly. He’s also an unabashed elitist, and goes on a hilarious pro-science rant at a PTA meeting that might not go over well in certain red corners of America, but whose reason and passion can’t be argued with.\nFrankenweenie doesn’t offer much in the way of new visual motifs; if you’ve seen a Burton film once or twice, you won’t be surprised by anything here. Still, the stop-motion animation is as beautiful as the medium has to offer. Shot in atmospheric black-and-white, the puppets come alive through variegated textures and unexpected angles. (The use of 3-D theatrics is surprisingly spare, so the $3 surcharge probably isn’t necessary to enjoy the film to its fullest.) \nBurton’s visual team at Disney has imagineered a carefully detailed world that weds Brancusi-like character design to recognizable laws of physics, creating a world that feels simultaneously familiar and strange. And for the first time in a long time, Burton’s obsessive love of creature features serves him well, with subtle, clever allusions to B-movies and recent thrillers. Best of all, the film’s morbidity is disgusting in a perversely joyous way, going beyond aestheticizing skeletons to mining for humor in the squirming, rotting beauty of life after life. \nThe frank simplicity of Frankenweenie is certainly crucial to its success, especially compared to the director’s overstuffed, convoluted Alice and Dark Shadows. All the same, it’s hard to resist wanting Burton to try just a little harder, especially when it comes to his lazy spoofing of the suburbs. The tale of a lonely misfit trying to escape the stultifying blandness of his pastel hell is a story Burton’s told time and again — done best in Edward Scissorhands, but also in Beetlejuice, Alice in Wonderland, and Corpse Bride. \nThe young Burton grew up in the ticky-tacky town of Burbank, California, and so it’s easy to understand why the director so often uses the ‘burbs as the straw man in his films. He wants his heroes to be mavericks, even progressives, polar opposites of characters like the traditional, cartoonishly patriarchal mayor in Frankenweenie who demands everyone be just like him. \nBut in the post-Douglas Sirk, post-Desperate Housewives, post-gentrification era of today, Burton’s “social satire” seems not only dated, but irrelevant. And too often, Burton’s gentle excoriation of the rigid social values of the suburbs serves as a cover for the fact that he is one of the most socially conservative mainstream filmmakers working today. His casts and characters are almost shockingly whitewashed, even for Hollywood, and his oeuvre brims full with passive, doll-eyed heroines in need of rescue. (Even the dragon-slaying Alice, Burton’s only female protagonist in thirty years of filmmaking, needed plenty of saving.) \nIn Frankenweenie’s New Holland, for example, the sole nonwhite resident is a vaguely villainous Japanese boy who fits all the negative stereotypes of Asian Americans: haughty, foreign, smart but uncreative. And when the students’ experiments go awry, Frankenweenie works up to a thrilling, satisfying climax, but the plot conspicuously twists and contorts to have its two male heroes take turns heroically saving their love interests. Tim Burton obviously revels in old-fashioned styles, but that’s no excuse for outdated sensibilities. \nInkoo Kang is a Boston-based film journalist and regular contributor to BoxOffice Magazine whose work has appeared in Pop Matters and Screen Junkies. She reviews stuff she hates, likes, and hate-likes on her blog THINK-O-VISION.\nFollow Inkoo Kang on Twitter.\nFollow Movieline on Twitter.', title: 'REVIEW: \'Frankenweenie,\' The Funny, Creepy — And Poignant — Tale of a Boy and His Undead Dog', published_at_raw: 'Fri, 05 Oct 2012 17:00:01 +0000', published_at: '2012-10-05T17:00:01.000Z', published_at_epoch: 1349456401000, created_at: '2012-10-17T10:44:01.161Z', created_at_epoch: 1350470641161, updated_at: '2012-10-17T10:44:01.161Z', updated_at_epoch: 1350470641161, source_id: 38298, source_type: 'rss', publisher_id: 11, guid: 'http://movieline.com/2012/10/05/review-frankenweenie-tim-burton-dog-asian-american/', guid_hash: '4026e7d6780a332c538c1ba048d4c609', url: 'http://movieline.com/2012/10/05/review-frankenweenie-tim-burton-dog-asian-american/', url_hash: '4026e7d6780a332c538c1ba048d4c609', url_normalized: 'http://movieline.com/2012/10/05/review-frankenweenie-tim-burton-dog-asian-american/', synapse_uuid: '7764fbc56e86b15451dbf1e3e11ff858', comments: 'http://movieline.com/2012/10/05/review-frankenweenie-tim-burton-dog-asian-american/#comments', media: [ { url: 'http://pmcmovieline.files.wordpress.com/2012/10/frankenweenie-review.jpg?w=120', medium: 'image', type: 'image/jpeg', title: 'frankenweenie-review', url_hash: 'bc9ead38b41e96f1a9d0edf54637e3c4', width: 120, height: 57, area: 6840 }, { url: 'http://0.gravatar.com/avatar/cd436d0e00baf2315e36f9bcc082e2f7?s=96&d=http%3A%2F%2F0.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96&r=PG', medium: 'image', type: null, title: 'bananabanfofana', url_hash: '9253fd8c30ae9e3156103f6d00fbcfe7', area: '9216', width: '96', height: '96' }, { url: 'http://pmcmovieline.files.wordpress.com/2012/10/frankenweenie-review.jpg', medium: 'image', url_hash: '3daca79bf8d6e9c90a558bf7abd507da', width: 630, height: 300, area: 189000, thumb_url: 'http://cdn.onswipestaging.com/entries/thumbs/4026e7d6780a332c538c1ba048d4c609_thumb.jpg', thumb_width: 630, thumb_height: 300, thumb_area: 189000 } ], video_media: [], featured_image: 'http://pmcmovieline.files.wordpress.com/2012/10/frankenweenie-review.jpg', featured_image_width: 630, featured_image_height: 300, featured_image_area: 189000, thumbnail_image: 'http://cdn.onswipestaging.com/entries/thumbs/4026e7d6780a332c538c1ba048d4c609_thumb.jpg', thumbnail_image_width: 630, thumbnail_image_height: 300, thumbnail_image_area: 189000, snippets: [] }

Thanks for your time.

--

Hello,

I assume you hit some buffer limit. Can you reproduce the issue with
single test documents if they're the same sort of size with the ones
you get in production? (maybe a few times bigger :D)

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Tue, Nov 6, 2012 at 2:21 AM, eldilibra ld@onswipe.com wrote:

Hey all. I've been trying to use Elasticsearch at work for 2 months now, but
keep running into the same problem. When indexing JSON objects, I get errors
like this:

{"error":"MapperParsingException[Failed to parse]; nested:
JsonParseException[Unexpected end-of-input within/between OBJECT entries\n
at [Source: [B@1dd6a67; line: 1, column: 36145]]; ","status":400}

Sometimes the text after "Unexpected end-of-input" is different, but it
doesn't matter, because the result is the same: the document is not indexed.
Now, to give you some background, I'm using a Node.js Elasticsearch client
to send these JSON objects, but I'm sure that is not the problem, since I
don't get these errors when using simple test objects (2 months of debugging
has led to my brazen certainty :~)). The documents I'm trying to index from
another production data store (MongoDB) are big, complex JSON docs. I know
that if the doc is different from the mapping, i'll have issues, but there
are certain fields in our docs that may be either strings or arrays. So, I
added index.mapping.ignore_malformed to my elasticsearch.yml (I'm using the
RC version 0.19.11 of ES, to take advantage of that ignore_malformed
feature). I thought that would take care of things, but alas, it hasn't. So
then I thought, Maybe I should encode these docs in Base64 to avoid
non-standard character problems that may be happening, since this
end-of-input thing could be caused by that sort of issue. However, I notice
that on the elasticsearch.org documentation site, Base64 encoding is only
mentioned when talking about the attachments plugin (allowing for .pdf,
.doc, etc.). Since I'm just indexing JSON though, that doesn't seem to be
answer, but I'm uncertain. Is there any benefit to encoding JSON in Base64
for ES, and if so, should I just do that then index it normally, without
using the attachment plugin? I'm just looking for some guidance. I apologize
if this post is all over the place, but I'm at my wit's end with a product
that I was very excited about getting into production, but have been unable
to.

An example of one of our documents:

--