Help updating index for 6.*

Elastic Stack Elasticsearch
1

A silent upgrade from 5.3 to 6.2 has killed Elasticsearch in an application due to the new structure of indices.

The existing (now wrong) format has multiple types, which I'm told by the error messages is not supported. However, the documentation states that multiple types are supported. After umpteen attempts, I have no idea what Elasticsearch wants from the index and time is fast running out.

As I understand it, _default_ is for the shared fields, while my_type is for type-specific fields.

Some advice here would be much appreciated.

2

A silent upgrade from 5.3 to 6.2

How this can happen? This is so bad.

At least you should have upgraded to latest 5.6 which would have tell you that you are using deprecated APIs.

3

Hi @dadoonet, this is the same problem we've been discussing in another thread.

The application must have multiple types, or it won't work.

4

I know. I'm asking how this happened? I mean there are 2 problems here:

5

Apologies for the length of the code, but I have an example of the settings and mappings for 5.3.

Using: PUT localhost:9200/asset_en_v1

{
	"settings": {
		"analysis": {
			"char_filter": {
				"&_to_and": {
					"type": "mapping",
					"mappings": ["&=> and "]
				}
			},
			"filter": {
				"asset_en_stopwords": {
					"type": "stop",
					"stopwords": ["_english_"]
				},
				"asset_en_stemmer": {
					"type": "stemmer",
					"name": "english"
				},
				"asset_en_shingle": {
					"type": "shingle",
					"max_shingle_size": 5,
					"min_shingle_size": 2,
					"output_unigrams": false,
					"output_unigrams_if_no_shingles": true
				}
			},
			"analyzer": {
				"asset_en_analyzer": {
					"type": "custom",
					"char_filter": ["html_strip", "&_to_and"],
					"tokenizer": "standard",
					"filter": [ "asset_en_stopwords", "asset_en_stemmer", "lowercase", "asset_en_shingle", "asciifolding" ]
				}
			}
		}
	},
	"mappings": {
		"note": {
			"_all": {
				"enabled": false
			},
			"properties": {
				"user_id": {
					"type": "long"
				},
				"creation": {
					"type": "date",
					"format": "date_hour_minute_second"
				},
				"deleted": {
					"type": "integer"
				},
				"favourite": {
					"type": "integer"
				},
				"modification": {
					"type": "date",
					"format": "date_hour_minute_second"
				},
				"note": {
					"type": "text",
					"analyzer": "english",
					"fields": {
						"std": {
							"type": "text",
							"analyzer": "asset_en_analyzer",
							"fields": {
								"std": {
									"type": "text",
									"analyzer": "standard"
								}
							}
						}
					}
				},
				"title": {
					"type": "text",
					"analyzer": "english",
					"fields": {
						"std": {
							"type": "text",
							"analyzer": "asset_en_analyzer",
							"fields": {
								"std": {
									"type": "text",
									"analyzer": "standard"
								}
							}
						}
					}
				},
				"links_to_asset": {
					"type": "nested",
					"properties": {
						"note_link_id": {
							"type": "long"
						},
						"user_id": {
							"type": "long"
						},
						"creation": {
							"type": "date",
							"format": "date_hour_minute_second"
						},
						"modification": {
							"type": "date",
							"format": "date_hour_minute_second"
						},
						"to_asset": {
							"type": "integer"
						},
						"from_asset": {
							"type": "integer"
						},
						"comment": {
							"type": "text",
							"fields": {
								"std": {
									"type": "text",
									"analyzer": "asset_en_analyzer",
									"fields": {
										"std": {
											"type": "text",
											"analyzer": "standard"
										}
									}
								}
							}
						}
					}
				}
			}
		},
		"folder": {
			"_all": {
				"enabled": false
			},
			"properties": {
				[ DUPLICATE PROPERTIES FOUND IN "NOTE" ],
				"in_folder_id": {
					"type": "long"
				},
				"label": {
					"type": "integer"
				}
			}
		},
		"bookmark": {
			"_all": {
				"enabled": false
			},
			"properties": {
				[ DUPLICATE PROPERTIES FOUND IN "NOTE" ],
				"url": {
					"type": "text",
					"index": "not_analyzed"
				},
				"publication_date": {
					"type": "date",
					"format": "date_hour_minute_second"
				}
			}
		},
		"message": {
			"_all": {
				"enabled": false
			},
			"properties": {
				[ DUPLICATE PROPERTIES FOUND IN "NOTE" ],
				"from": {
					"type": "text",
					"index": "not_analyzed"
				},
				"seen": {
					"type": "text",
					"index": "not_analyzed"
				}
			}
		}
	}
}
1 Like
6

Did you see this blog post? https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch

It contains a lot of information about moving from multiple types to single type.

7

Yes, I have, and I can't get it to work.

Also, if I could get it to work I would end up with multiple index types, which would require a massive refactoring of code.

As an example, PUT localhost:9200/asset_en_v1 creates a single index, which I then create an alias to, using that in the application.

If I use the example in the article, I would have an index per type, which — as I said — would require a massive refactoring of code.

Would I need a separate settings per each type that would now have become an index? I have no idea, and the documentation offers no guide I'm able to find.

8

Here's another example:

$document = $this->elasticsearch_model->getIndex([
	'index' => "asset",
	'type' => $this->get_type($note),
	'id' => $note->note_id
]);

If I made the type the index, what would the type become in the example above?

I would have to change each instance of that code, and there are thousands.

9

I really think you need to change your code.
Or you can setup a cluster with 5.6, reindex what you had to fix the short term issue.

Note that if you are not creating new indices in 6.x but just reading 5.x indices, multiple types are supported.
If you are creating new indices, then you need to change your application I'm afraid.

10

Hi @dadoonet...

Note that if you are not creating new indices in 6.x but just reading 5.x indices, multiple types are supported.

I first noticed there was a problem when I got the error:

{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}

I hadn't made changes to either the application, or to the schema of Elasticsearch. So no, multiple types are not supported, nor anything else in 5.x under 6.x.

I still don't understand the difference between an index and a type if multiple types aren't supported.

Also, more than anything else, I need some guidance on what changes I need to make to the schema, based on the existing example I've shared — I've been working on this for weeks and NOTHING works.

11

That error message is often an indication that you are running out of disk space and have exceeded the high disk usage watermarks.

12

Hi @Christian_Dahlqvist, I have gigabytes of free space, and the cause wasn't that but the fact that Elasticsearch had upgraded to 6 without me knowing.

13

The error you are seeing is unrelated to the multiple types per index problem.

So no, multiple types are not supported, nor anything else in 5.x under 6.x.

Yes they are.

I still don't understand the difference between an index and a type if multiple types aren't supported.

That's why we are removing the concept of types. Once this will be gone, you won't have to think about it anymore.
It used to be there from the beginning of Elasticsearch but as explained in the article I linked to, it was not a so good idea at the end of the day.

14

Hi @dadoonet, please read the response to @Christian_Dahlqvist.

15

That's why we are removing the concept of types. Once this will be gone, you won't have to think about it anymore.

So I don't have to use the types field?

16

How large percentage of the disk is available? The flooding stage is set at 95% of capacity, so if you have large disks you could be exceeding this even if you have gigabytes of space left.

17

It's above the percentage stated in the documentation.

As I said, Elasticsearch had upgraded itself.

While attempting to migrate from 5.3 to 6.x I've since deleted the schema and the data, so there's no chance of returning to that, as I get errors from 6.x when attempting to add the previous schema.

Is it possible to downgrade?

18

Then I believe that most likely is the cause of the error you listed.

19

You must use the type field in the URL as it's still there. Use _doc as the type name.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.