Hello everyone,
I have been working with the dynamic mapping of Elasticsearch for some time
now and so far dynamic mapping worked fine: the schema of my documents
differ a lot with respect to the type. For instance, sometimes a certain
field is of type object and sometimes of type array. As far as I know that
is no problem, there will be a mapping issue, but the document gets indexed
anyway. That was the case until now.
Not anymore. I use Elasticsearch v0.9.0 The problem in this thread of the
mailing list describes the similar problem
http://elasticsearch-users.115913.n3.nabble.com/Having-trouble-with-MappingParserExceptions-any-ideas-td4024909.html
It hints that the reason could be the buffer? Unfortunately there is no
answer. Errors are like the following
{"error":"MapperParsingException[failed to parse [record.resources.hash]];
nested: ElasticSearchIllegalArgumentException[unknown property [content]];
","status":400}
It seems fine, because apparently the mapping is not compatible here. But
so far ES was always able to index it anyway. I have written a cURL
recreation
The result is:
{"error":"MapperParsingException[failed to parse [record.date]]; nested:
MapperParsingException[failed to parse date field [}], tried both date
format [dateOptionalTime], and timestamp number]; nested:
IllegalArgumentException[Invalid format: "}"]; ","status":400}
Which I do not understand, as there is no record.date field. Moreover it
seems like this is invalid JSON, but it is not. I have parsed it through
several JSON parser.
Is there a way to index the documents anyway? Can I circumvent the mapping,
as in deactivate the mapping exceptions? What other approach can I take
here, I am really stuck.
Best,
Konrad
cURL recreation
Here is my command (optional Gist including formatting
https://gist.github.com/platzhirsch/96e76b95e8a308646e62)
$ curl -XDELETE "http://localhost:9200/metadata/ckan"
$ curl -XDELETE "http://localhost:9200/metadata/"
$ curl -XPOST "http://localhost:9200/metadata/ckan" -d '{
"date": "2013-09-17",
"record": {
"author": "URSEA",
"author_email": "consultas@ursea.gub.uy",
"extras": [
{
"id": "20b6d7b5-7c8b-4ff7-88ba-1b42c6bddc04",
"key": "spatial_coverage",
"package_id": "e04d4159-6839-4d04-a110-a5a300886ed6",
"revision_id": "3bf6b52b-fb19-41a9-a826-eb055071ba90",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"value": ""
},
{
"id": "39ce2c72-acd3-4da9-9ab8-ad55e061464c",
"key": "spatial_ref_system",
"package_id": "e04d4159-6839-4d04-a110-a5a300886ed6",
"revision_id": "3bf6b52b-fb19-41a9-a826-eb055071ba90",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"value": ""
},
{
"id": "5a755c45-a47b-46b4-966a-a55095a672ff",
"key": "temporal_coverage",
"package_id": "e04d4159-6839-4d04-a110-a5a300886ed6",
"revision_id": "3bf6b52b-fb19-41a9-a826-eb055071ba90",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"value": ""
},
{
"id": "9334c8ca-d2c6-4a08-9b3b-51739e5a3fa1",
"key": "update_frequency",
"package_id": "e04d4159-6839-4d04-a110-a5a300886ed6",
"revision_id": "3bf6b52b-fb19-41a9-a826-eb055071ba90",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"value": "15"
}
],
"groups": [],
"id": "e04d4159-6839-4d04-a110-a5a300886ed6",
"isopen": false,
"license_id": "odc-uy",
"license_title": "Uruguay Open Data Licence",
"maintainer": "Susana Masoller",
"maintainer_email": "susana.masoller@ursea.gub.uy",
"metadata_created": "2013-06-20T13:38:30.011961",
"metadata_modified": "2013-09-09T18:38:47.827532",
"name": "autorizaciones-rspebt",
"notes": "Productos de baja tensi\u00f3n autorizados para su
comercializaci\u00f2n en el marco del Reglamento de Seguridad de Productos
El\u00e9ctricos de Baja Tensi\u00f3n ",
"num_resources": 6,
"num_tags": 3,
"organization": {
"approval_status": "approved",
"created": "2013-06-06T14:44:11.035837",
"description": "Unidad Reguladora de Servicios de Energ\u00eda
y Agua",
"id": "e846db7c-eada-46c4-927a-439126915d39",
"image_url":
"http://peu.agesic.gub.uy/BancoDeImagenes/galleries/Datos/URSEA.jpg",
"is_organization": true,
"name": "ursea",
"revision_id": "b80102c2-3b9d-4039-9d44-502a1b87017e",
"revision_timestamp": "2013-06-21T20:16:30.682337",
"state": "active",
"title": "URSEA",
"type": "organization"
},
"owner_org": "e846db7c-eada-46c4-927a-439126915d39",
"private": false,
"relationships_as_object": [],
"relationships_as_subject": [],
"resources": [
{
"cache_last_updated": null,
"cache_url": null,
"created": "2013-06-21T13:09:16.356437",
"description": "Productos el\u00e9ctricos autorizados para
su comercializaci\u00f3n - RSPEBT",
"format": "CSV",
"hash": {
"content": "6fd94b6dea5211c33b456546893b1c540b31017d",
"header": "eef66ac8ee15bd9d01f2b2e5679977db279fb62a"
},
"id": "c2fc339a-dff7-4882-bab8-03552c99d718",
"last_modified": "2013-07-22T18:27:23.789945",
"mimetype": "text/csv",
"mimetype_inner": null,
"name": "Base Autorizaciones - 21-06-2013-para
publicar.csv",
"position": 0,
"resource_group_id": "bf777e9f-13db-458f-bd88-e5561ab4c31a",
"resource_type": "file.upload",
"revision_id": "905464db-65b6-40df-a92f-f8296143b92b",
"revision_timestamp": "2013-07-22T21:27:23.878752",
"size": "2674108",
"state": "active",
"tracking_summary": {
"recent": 0,
"total": 0
},
"url":
"http://catalogodatos.gub.uy/storage/f/2013-06-21T16%3A11%3A41.609Z/base-autorizaciones-21-06-2013-para-publicar.csv",
"webstore_last_updated": null,
"webstore_url": null
},
{
"cache_last_updated": null,
"cache_url": null,
"created": "2013-07-09T10:40:07.439787",
"description": "",
"format": "CSV",
"hash": {
"content": "f7216c9264f8acc6548b96092afe84443dc26d4c",
"header": "97924dc5378bfed9513a209231de2837024b86ce"
},
"id": "e5d72a22-4e42-4750-81fa-e7f04115681b",
"last_modified": "2013-07-22T18:27:24.353776",
"mimetype": "text/csv",
"mimetype_inner": null,
"name": "Base Certificados - 09-07-2013-para publicar.csv",
"position": 1,
"resource_group_id": "bf777e9f-13db-458f-bd88-e5561ab4c31a",
"resource_type": "file.upload",
"revision_id": "d91c2279-4f35-413f-9b1e-c60d9963a59b",
"revision_timestamp": "2013-07-22T21:27:24.425180",
"size": "2671750",
"state": "active",
"tracking_summary": {
"recent": 0,
"total": 0
},
"url":
"http://catalogodatos.gub.uy/storage/f/2013-07-09T13%3A41%3A34.031Z/base-certificados-09-07-2013-para-publicar.csv",
"webstore_last_updated": null,
"webstore_url": null
},
{
"cache_last_updated": null,
"cache_url": null,
"created": "2013-07-15T16:34:35.403672",
"description": "",
"format": "CSV",
"hash": {
"content": "d3e262b9a739f5e9e5baf968773d5f5581e800d1",
"header": "37b0e4315381ffe80a0c85cd964d74da3f2e5f01"
},
"id": "96453432-8381-42c8-b125-3e32e124e09b",
"last_modified": "2013-07-22T18:27:24.738445",
"mimetype": "text/csv",
"mimetype_inner": null,
"name": "Autorizaciones vigencia 15-07-2013",
"position": 2,
"resource_group_id": "bf777e9f-13db-458f-bd88-e5561ab4c31a",
"resource_type": "file.upload",
"revision_id": "0f8247e5-f137-42c7-ae8e-1fbdb5b1f251",
"revision_timestamp": "2013-07-22T21:27:24.881333",
"size": "2690443",
"state": "active",
"tracking_summary": {
"recent": 0,
"total": 0
},
"url":
"http://catalogodatos.gub.uy/storage/f/2013-07-15T19%3A35%3A51.046Z/base-certificados-15-07-2013-para-publicar.csv",
"webstore_last_updated": null,
"webstore_url": null
},
{
"cache_last_updated": null,
"cache_url": null,
"created": "2013-08-12T13:54:24.101737",
"description": "",
"format": "CSV",
"hash": "",
"id": "aed4eefd-c3b8-4ecf-b5e2-c3bd907ee1f2",
"last_modified": null,
"mimetype": null,
"mimetype_inner": null,
"name": "Base Certificados - 12-08-2013-para publicar.csv",
"position": 3,
"resource_group_id": "bf777e9f-13db-458f-bd88-e5561ab4c31a",
"resource_type": "file.upload",
"revision_id": "3db1d492-78b2-4181-96e9-450ece7607da",
"revision_timestamp": "2013-08-12T16:54:24.043877",
"size": null,
"state": "active",
"tracking_summary": {
"recent": 0,
"total": 0
},
"url":
"http://catalogodatos.gub.uy/storage/f/2013-08-12T16%3A55%3A53.078Z/base-certificados-12-08-2013-para-publicar.csv",
"webstore_last_updated": null,
"webstore_url": null
},
{
"cache_last_updated": null,
"cache_url": null,
"created": "2013-09-03T14:39:20.876073",
"description": "",
"format": "CSV",
"hash": "",
"id": "9bd08819-d38e-4dda-a5c5-e305b83007dd",
"last_modified": null,
"mimetype": null,
"mimetype_inner": null,
"name": "Autorizaciones - 03-09-2013-para web.csv",
"position": 4,
"resource_group_id": "bf777e9f-13db-458f-bd88-e5561ab4c31a",
"resource_type": "file.upload",
"revision_id": "1275dbf6-f99a-46a3-9a97-aa070c876498",
"revision_timestamp": "2013-09-03T17:39:20.824951",
"size": null,
"state": "active",
"tracking_summary": {
"recent": 0,
"total": 0
},
"url":
"http://catalogodatos.gub.uy/storage/f/2013-09-03T17%3A40%3A51.062Z/autorizaciones-03-09-2013-para-web.csv",
"webstore_last_updated": null,
"webstore_url": null
},
{
"cache_last_updated": null,
"cache_url": null,
"created": "2013-09-09T15:38:47.919014",
"description": "",
"format": "CSV",
"hash": "",
"id": "73d00ea6-d752-433f-93ee-4b4131609c32",
"last_modified": null,
"mimetype": null,
"mimetype_inner": null,
"name": "Base Certificados - 09-09-2013-para publicar.csv",
"position": 5,
"resource_group_id": "bf777e9f-13db-458f-bd88-e5561ab4c31a",
"resource_type": "file.upload",
"revision_id": "9c5e476d-8c79-4416-b55d-01e840299480",
"revision_timestamp": "2013-09-09T18:38:47.827532",
"size": null,
"state": "active",
"tracking_summary": {
"recent": 0,
"total": 0
},
"url":
"http://catalogodatos.gub.uy/storage/f/2013-09-09T18%3A40%3A23.968Z/base-certificados-09-09-2013-para-publicar.csv",
"webstore_last_updated": null,
"webstore_url": null
}
],
"revision_id": "d30f283f-d59b-4c45-8c23-e6e3a4352edd",
"revision_timestamp": "2013-06-20T13:41:38.244339",
"state": "active",
"tags": [
{
"display_name": "energ\u00eda",
"id": "4dac9c7c-a715-43c7-b6cf-c5c15cf62586",
"name": "energ\u00eda",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"vocabulary_id": null
},
{
"display_name": "productos autorizados",
"id": "fad95763-e2b5-4a36-a49f-a20fae67ae11",
"name": "productos autorizados",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"vocabulary_id": null
},
{
"display_name": "seguridad el\u00e9ctrica",
"id": "794a04a3-c824-4b0e-ab5d-ee5096035789",
"name": "seguridad el\u00e9ctrica",
"revision_timestamp": "2013-06-20T13:38:30.011961",
"state": "active",
"vocabulary_id": null
}
],
"title": "autorizaciones RSPEBT",
"tracking_summary": {
"recent": 0,
"total": 0
},
"type": "dataset",
"url": null,
"version": null
},
"repository": "catalogodatos.gub.uy"
}'
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.