Help updating index for 6.*

waynesmallman · April 27, 2018, 9:31am

A silent upgrade from 5.3 to 6.2 has killed Elasticsearch in an application due to the new structure of indices.

The existing (now wrong) format has multiple types, which I'm told by the error messages is not supported. However, the documentation states that multiple types are supported. After umpteen attempts, I have no idea what Elasticsearch wants from the index and time is fast running out.

As I understand it, _default_ is for the shared fields, while my_type is for type-specific fields.

Some advice here would be much appreciated.

dadoonet · April 27, 2018, 9:56am

A silent upgrade from 5.3 to 6.2

How this can happen? This is so bad.

At least you should have upgraded to latest 5.6 which would have tell you that you are using deprecated APIs.

waynesmallman · April 27, 2018, 10:02am

Hi @dadoonet, this is the same problem we've been discussing in another thread.

The application must have multiple types, or it won't work.

dadoonet · April 27, 2018, 10:06am

I know. I'm asking how this happened? I mean there are 2 problems here:

Something upgraded your elasticsearch cluster without following the guides like: https://www.elastic.co/products/upgrade_guide
You need now to fix the problem caused by the previous one.

waynesmallman · April 27, 2018, 10:08am

Apologies for the length of the code, but I have an example of the settings and mappings for 5.3.

Using: PUT localhost:9200/asset_en_v1

{
	"settings": {
		"analysis": {
			"char_filter": {
				"&_to_and": {
					"type": "mapping",
					"mappings": ["&=> and "]
				}
			},
			"filter": {
				"asset_en_stopwords": {
					"type": "stop",
					"stopwords": ["_english_"]
				},
				"asset_en_stemmer": {
					"type": "stemmer",
					"name": "english"
				},
				"asset_en_shingle": {
					"type": "shingle",
					"max_shingle_size": 5,
					"min_shingle_size": 2,
					"output_unigrams": false,
					"output_unigrams_if_no_shingles": true
				}
			},
			"analyzer": {
				"asset_en_analyzer": {
					"type": "custom",
					"char_filter": ["html_strip", "&_to_and"],
					"tokenizer": "standard",
					"filter": [ "asset_en_stopwords", "asset_en_stemmer", "lowercase", "asset_en_shingle", "asciifolding" ]
				}
			}
		}
	},
	"mappings": {
		"note": {
			"_all": {
				"enabled": false
			},
			"properties": {
				"user_id": {
					"type": "long"
				},
				"creation": {
					"type": "date",
					"format": "date_hour_minute_second"
				},
				"deleted": {
					"type": "integer"
				},
				"favourite": {
					"type": "integer"
				},
				"modification": {
					"type": "date",
					"format": "date_hour_minute_second"
				},
				"note": {
					"type": "text",
					"analyzer": "english",
					"fields": {
						"std": {
							"type": "text",
							"analyzer": "asset_en_analyzer",
							"fields": {
								"std": {
									"type": "text",
									"analyzer": "standard"
								}
							}
						}
					}
				},
				"title": {
					"type": "text",
					"analyzer": "english",
					"fields": {
						"std": {
							"type": "text",
							"analyzer": "asset_en_analyzer",
							"fields": {
								"std": {
									"type": "text",
									"analyzer": "standard"
								}
							}
						}
					}
				},
				"links_to_asset": {
					"type": "nested",
					"properties": {
						"note_link_id": {
							"type": "long"
						},
						"user_id": {
							"type": "long"
						},
						"creation": {
							"type": "date",
							"format": "date_hour_minute_second"
						},
						"modification": {
							"type": "date",
							"format": "date_hour_minute_second"
						},
						"to_asset": {
							"type": "integer"
						},
						"from_asset": {
							"type": "integer"
						},
						"comment": {
							"type": "text",
							"fields": {
								"std": {
									"type": "text",
									"analyzer": "asset_en_analyzer",
									"fields": {
										"std": {
											"type": "text",
											"analyzer": "standard"
										}
									}
								}
							}
						}
					}
				}
			}
		},
		"folder": {
			"_all": {
				"enabled": false
			},
			"properties": {
				[ DUPLICATE PROPERTIES FOUND IN "NOTE" ],
				"in_folder_id": {
					"type": "long"
				},
				"label": {
					"type": "integer"
				}
			}
		},
		"bookmark": {
			"_all": {
				"enabled": false
			},
			"properties": {
				[ DUPLICATE PROPERTIES FOUND IN "NOTE" ],
				"url": {
					"type": "text",
					"index": "not_analyzed"
				},
				"publication_date": {
					"type": "date",
					"format": "date_hour_minute_second"
				}
			}
		},
		"message": {
			"_all": {
				"enabled": false
			},
			"properties": {
				[ DUPLICATE PROPERTIES FOUND IN "NOTE" ],
				"from": {
					"type": "text",
					"index": "not_analyzed"
				},
				"seen": {
					"type": "text",
					"index": "not_analyzed"
				}
			}
		}
	}
}

dadoonet · April 27, 2018, 10:10am

Did you see this blog post? https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch

It contains a lot of information about moving from multiple types to single type.

waynesmallman · April 27, 2018, 10:17am

Yes, I have, and I can't get it to work.

Also, if I could get it to work I would end up with multiple index types, which would require a massive refactoring of code.

As an example, PUT localhost:9200/asset_en_v1 creates a single index, which I then create an alias to, using that in the application.

If I use the example in the article, I would have an index per type, which — as I said — would require a massive refactoring of code.

Would I need a separate settings per each type that would now have become an index? I have no idea, and the documentation offers no guide I'm able to find.

waynesmallman · April 27, 2018, 10:21am

Here's another example:

$document = $this->elasticsearch_model->getIndex([
	'index' => "asset",
	'type' => $this->get_type($note),
	'id' => $note->note_id
]);

If I made the type the index, what would the type become in the example above?

I would have to change each instance of that code, and there are thousands.

dadoonet · April 27, 2018, 10:42am

I really think you need to change your code.
Or you can setup a cluster with 5.6, reindex what you had to fix the short term issue.

Note that if you are not creating new indices in 6.x but just reading 5.x indices, multiple types are supported.
If you are creating new indices, then you need to change your application I'm afraid.

waynesmallman · May 5, 2018, 10:00am

Hi @dadoonet...

Note that if you are not creating new indices in 6.x but just reading 5.x indices, multiple types are supported.

I first noticed there was a problem when I got the error:

{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}

I hadn't made changes to either the application, or to the schema of Elasticsearch. So no, multiple types are not supported, nor anything else in 5.x under 6.x.

I still don't understand the difference between an index and a type if multiple types aren't supported.

Also, more than anything else, I need some guidance on what changes I need to make to the schema, based on the existing example I've shared — I've been working on this for weeks and NOTHING works.

Christian_Dahlqvist · May 5, 2018, 10:03am

That error message is often an indication that you are running out of disk space and have exceeded the high disk usage watermarks.

waynesmallman · May 5, 2018, 10:04am

Hi @Christian_Dahlqvist, I have gigabytes of free space, and the cause wasn't that but the fact that Elasticsearch had upgraded to 6 without me knowing.

dadoonet · May 5, 2018, 10:05am

The error you are seeing is unrelated to the multiple types per index problem.

So no, multiple types are not supported, nor anything else in 5.x under 6.x.

Yes they are.

I still don't understand the difference between an index and a type if multiple types aren't supported.

That's why we are removing the concept of types. Once this will be gone, you won't have to think about it anymore.
It used to be there from the beginning of Elasticsearch but as explained in the article I linked to, it was not a so good idea at the end of the day.

waynesmallman · May 5, 2018, 10:05am

Hi @dadoonet, please read the response to @Christian_Dahlqvist.

waynesmallman · May 5, 2018, 10:07am

That's why we are removing the concept of types. Once this will be gone, you won't have to think about it anymore.

So I don't have to use the types field?

Christian_Dahlqvist · May 5, 2018, 10:07am

How large percentage of the disk is available? The flooding stage is set at 95% of capacity, so if you have large disks you could be exceeding this even if you have gigabytes of space left.

waynesmallman · May 5, 2018, 10:08am

It's above the percentage stated in the documentation.

As I said, Elasticsearch had upgraded itself.

While attempting to migrate from 5.3 to 6.x I've since deleted the schema and the data, so there's no chance of returning to that, as I get errors from 6.x when attempting to add the previous schema.

Is it possible to downgrade?

Christian_Dahlqvist · May 5, 2018, 10:11am

Then I believe that most likely is the cause of the error you listed.

dadoonet · May 5, 2018, 10:13am

You must use the type field in the URL as it's still there. Use _doc as the type name.

system · June 2, 2018, 10:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Allowing multiple mappings in Elastic Search 6 Elasticsearch	6	1678	March 1, 2021
What must I do with the "type" feature that not available anymore when Migrating 5.6.2 to Latest Stable Elasticsearch	3	358	September 6, 2019
Creating new types in 6.0 Elasticsearch	6	1455	January 9, 2018
Any place to get more details about upgrading from Es 2.x to ES 6.x? Elasticsearch	4	594	February 8, 2018
Updated 5.6,X to 6.2 and all my data is gone Elasticsearch	4	1228	March 7, 2018

Help updating index for 6.*

Related topics