Import data failed document - Some documents could not be imported

Elastic Stack Kibana
1

I am in the localhost:5601 GUI on the "Add data / Upload File" page.

I am importing a ndjson file with 86 lines and the following mappings set in "Import data / Advanced"

"mappings": {
  {
    "properties": {
      "site": {
        "type": "text"
      },
      "geometry": {
        "type": "geo_shape"
        }
      }
  }
}

When I create my index name and hit "import" I get the warning:
"1 out of 86 documents could not be imported. This could be due to lines not matching the Grok pattern."

When I try to expand the "Failed documents", nothing gets shown...

How can I efficiently debug / investigate the failing document?

2

Do you see any errors in logs or in browser console?

Thanks,
Bhavya

4

@bhavyarm another run with "test" index and full docker logs since clicking import

kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:56+00:00","tags":[],"pid":1217,"method":"get","statusCode":200,"req":{"url":"/internal/file_upload/has_import_permission?checkCreateIndexPattern=true&checkHasManagePipeline=true&indexName=test","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":7,"contentLength":28},"message":"GET /internal/file_upload/has_import_permission?checkCreateIndexPattern=true&checkHasManagePipeline=true&indexName=test 200 7ms - 28.0B"}
elastic_search   | {"type": "server", "timestamp": "2021-11-09T23:26:57,834Z", "level": "INFO", "component": "o.e.c.m.MetadataCreateIndexService", "cluster.name": "docker-cluster", "node.name": "f63b0da926aa", "message": "[test] creating index, cause [api], templates [], shards [1]/[1]", "cluster.uuid": "ohvpaAn_QqWkrQ2hrD5mEw", "node.id": "4XmFjIPERPCZ5vYulArxHw"  }
kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:57+00:00","tags":[],"pid":1217,"method":"post","statusCode":200,"req":{"url":"/internal/file_upload/import","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","origin":"http://localhost:5601","content-length":"246","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":649,"contentLength":340},"message":"POST /internal/file_upload/import 200 649ms - 340.0B"}
kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:58+00:00","tags":[],"pid":1217,"method":"post","statusCode":200,"req":{"url":"/internal/file_upload/import?id=avzbo0iaq","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","origin":"http://localhost:5601","content-length":"836655","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":881,"contentLength":87292},"message":"POST /internal/file_upload/import?id=avzbo0iaq 200 881ms - 85.2KB"}
kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:59+00:00","tags":[],"pid":1217,"method":"get","statusCode":200,"req":{"url":"/api/index_patterns/_fields_for_wildcard?pattern=test&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":12,"contentLength":969},"message":"GET /api/index_patterns/_fields_for_wildcard?pattern=test&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 12ms - 969.0B"}
kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:59+00:00","tags":[],"pid":1217,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/_find?fields=title&per_page=10&search=%22test%22&search_fields=title&type=index-pattern","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":11,"contentLength":53},"message":"GET /api/saved_objects/_find?fields=title&per_page=10&search=%22test%22&search_fields=title&type=index-pattern 200 11ms - 53.0B"}
kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:59+00:00","tags":[],"pid":1217,"method":"post","statusCode":200,"req":{"url":"/api/saved_objects/index-pattern","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","origin":"http://localhost:5601","content-length":"102","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":645,"contentLength":357},"message":"POST /api/saved_objects/index-pattern 200 645ms - 357.0B"}
kibana           | {"type":"response","@timestamp":"2021-11-09T23:26:59+00:00","tags":[],"pid":1217,"method":"get","statusCode":200,"req":{"url":"/api/index_patterns/_fields_for_wildcard?pattern=test&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":13,"contentLength":969},"message":"GET /api/index_patterns/_fields_for_wildcard?pattern=test&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 13ms - 969.0B"}
kibana           | {"type":"response","@timestamp":"2021-11-09T23:27:37+00:00","tags":[],"pid":1217,"method":"post","statusCode":200,"req":{"url":"/api/ui_counters/_report","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/home","content-type":"application/json","kbn-version":"7.14.2","kbn-system-request":"true","origin":"http://localhost:5601","content-length":"153","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","dnt":"1","sec-gpc":"1"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","referer":"http://localhost:5601/app/home"},"res":{"statusCode":200,"responseTime":656,"contentLength":15},"message":"POST /api/ui_counters/_report 200 656ms - 15.0B"}
5

@bhavyarm good suggestion on using browser console, I have discovered the problem document! Will investigate and let you know the issue with it.

{
	"success": true,
	"id": "kpyyjga17",
	"index": "blah",
	"docCount": 86,
	"failures": [
		{
			"item": 38,
			"reason": "failed to parse field [geometry] of type [geo_shape]",
			"doc": "{\"OBJECTID\": 39, \"searchable\": \"field value example\", \"geometry\": {\"type\": \"multipolygon\", \"coordinates\": [[[[etc....]]]]}}"
		}
	]
}

But the reason is:
"failed to parse field [geometry] of type [geo_shape]

I'm also noticing in the browser console that on the POST import response in network tab, that I get a status:400 type: "parse_exception", reason: "[processors] required property is missing"
So I will investigate that too

6

@bhavyarm I have found the issue. That particular document had a multipolygon that was self-intersecting as shown in the image. The failure message could be a bit more transparent though!
Geo_self_intersection

7

Great. Thanks for keeping us posted.

I totally agree that error message can be improved. Will log a bug.

Thanks,
Bhavya

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.