# Elasticsearch index contains documents where source is invalid json

**URL:** <https://discuss.elastic.co/t/elasticsearch-index-contains-documents-where-source-is-invalid-json/2010>\
**Category:** Elasticsearch\
**Created:** [June 5, 2015, 1:35pm UTC](https://discuss.elastic.co/t/elasticsearch-index-contains-documents-where-source-is-invalid-json/2010 "2015-06-05T13:35:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![stubbetje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stubbetje/32/534_2.png) [@stubbetje](https://discuss.elastic.co/u/stubbetje)\
**Post date:** [June 5, 2015, 1:35pm UTC](https://discuss.elastic.co/t/elasticsearch-index-contains-documents-where-source-is-invalid-json/2010/1 "2015-06-05T13:35:34Z")

</div>

I'm in the process of migrating a complete index from one cluster to another.  
I'm selecting all the documents from the old cluster with the scroll api and bulk indexing them to the new cluster, using elasticsearch-php.

The issue I'm currently having is that while "scrolling" through the documents on the old cluster, I am getting JsonDeserializationError (php) exception with the message "Syntax error".

It took me some time to figure out the root cause of this exception, but apparently I have documents in the old cluster off which the contents is not valid json.

A simplyfied format of the document is:

```
  { { /* first json object */ } { /* second json object */ } { /* second json object */ } }

```

(note that there is no comma (`,`) between the _inner objects_.

I have no idea how this kind of invalid document could ever have been indexed in ES. I have serveral of these types of documents, all dating from January 2015.

This particular index has survived a few elasticsearch upgrades, but not sure which version of ES I was using at the moment the documents have been indexed, but according the released dates it could not have been anything later than version 1.4.2.

I was really surprised that this could ever happen, but is probably related to a bug that occurred in an older version.

Any way, this makes moving my index to the new cluster quite troublesome:

- The elasticsearch-php libary tries to parse the ES response and throws an exception
- I could catch the exception, but there is no easy way to know which document caused this
- I could set the page size to 1 document and then I could just ignore the exception, but does not really seem very efficient on resources.

Is there a quick way to know which documents contain invalid json?  
Or a kind of fsck process to detect these kind of documents?

I am currently making a PHP script for selecting all the items, but doing so without using the elasticsearch-php library, because the library does not provide an option to just return the document as a string.

---

<div class="post-metadata">

**Author:** ![balu\_giduturi](https://avatars.discourse-cdn.com/v4/letter/b/34f0e0/32.png) [@balu\_giduturi](https://discuss.elastic.co/u/balu_giduturi)\
**Post date:** [February 11, 2016, 1:38pm UTC](https://discuss.elastic.co/t/elasticsearch-index-contains-documents-where-source-is-invalid-json/2010/2 "2016-02-11T13:38:56Z")

</div>

Hi Stefan,

I also faced the same problem, the issue is :  
scroll id malfunctioned don't know how

i tried to overcome this by exceptional handling (try catch) below which worked for me;

```
$docs = $_client->search($searchparams);
$scroll_id = $docs['_scroll_id'];
 while (\true) {
 try {
	$response = $_client->scroll(
		array(
		    "scroll_id" => $scroll_id,
		    "scroll" => "20m"
		)
	);
	$scroll_id = $response['_scroll_id'];

	// your stuff
}
catch (Exception $error) {
                echo "scroll_id :::::" . $scroll_id . " " . $error->getMessage();
            }
 }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:17pm UTC](https://discuss.elastic.co/t/elasticsearch-index-contains-documents-where-source-is-invalid-json/2010/3 "2017-07-05T23:17:11Z")

</div>