Upgrading ES from 1.7.1 to 2.x Issue

Elastic Stack Elasticsearch
22

It seems that some files are missing.

You should have something in:

/data/elasticsearch/adviqo-restore/nodes/0/indices/index_31/_state

But apparently it's not there.

The only reason we can imagine is that you were using dedicated master nodes. In that case, the _state dir does not live within data nodes but in master nodes.
Can you confirm you are using dedicated master nodes?

23

Hi,

All nodes where able to become master by config and in every index directory there is a filled _state directory

It's like all data is there but corrupt or else.

24

Can you list one of the _state dir (ls -l)?

25

in every directory exists one state-34.st file (didget differs)

26

Can you change gateway: TRACE in logging.yml?
And check in logs if you have something like found state file:

27

There are some state files found, unfortunately none of my needed indices.

28

Can you paste here some of those lines?

I'm reading the source code again and I don't understand how this could happen.

                final Path stateDir = dataLocation.resolve(STATE_DIR_NAME);
                // now, iterate over the current versions, and find latest one
                // we don't check if the stateDir is present since it could be deleted
                // after the check. Also if there is a _state file and it's not a dir something is really wrong
                try (DirectoryStream<Path> paths = Files.newDirectoryStream(stateDir)) { // we don't pass a glob since we need the group part for parsing
                    for (Path stateFile : paths) {
                        final Matcher matcher = stateFilePattern.matcher(stateFile.getFileName().toString());
                        if (matcher.matches()) {
                            final long stateId = Long.parseLong(matcher.group(1));
                            maxStateId = Math.max(maxStateId, stateId);
                            final boolean legacy = MetaDataStateFormat.STATE_FILE_EXTENSION.equals(matcher.group(2)) == false;
                            maxStateIdIsLegacy &= legacy; // on purpose, see NOTE below
                            PathAndStateId pav = new PathAndStateId(stateFile, stateId, legacy);
                            logger.trace("found state file: {}", pav);
                            files.add(pav);
                        }
                    }
                } catch (NoSuchFileException | FileNotFoundException ex) {
                    // no _state directory -- move on
                }

stateDir here is _state. We check for files like state-XX.st which sounds to exist here. So logger.trace("found state file: {}", pav); should print it...

Are those files readable by the elasticsearch user which is running the elasticsearch process?

What gives ls -l /index_53/1/_state for example? And what gives the same command for an index which is loaded correctly by elasticsearch?

29

an ls -l at

/data/elasticsearch/adviqo-restore/nodes/0/indices/index_53/0/_state

prints

-rw-r--r-- 1 elasticsearch elasticsearch 85 30. Nov 13:41 state-29290.st

an working index like .../.marvel-es-2015.12.07/_state

looks like

-rw-r--r-- 1 elasticsearch elasticsearch 6013 7. Dez 12:07 state-4.st

the difference i find is like in the working ste file is way more (barely readable) information than in the not working indices.

for the working indices it's just says :

found state file: [id:3, legacy:false, file:/data/elasticsearch/adviqo-restore/nodes/0/indices/.marvel-es-2015.12.07/_state/state-3.st]

other not working indices are not even mentioned, or just like:

][DEBUG][gateway ] [Krakkan] [index_68] failed to find metadata for existing index location

and :

][DEBUG][gateway ] [Krakkan] [index_68] dangling index directory detected, but no state found

30

state-29290.st file seems to be super small indeed.

Also I can see that you first post happened in Dec 1st. The date of this latest .st file is 30. Nov 13:41.
Any chance you recall what you did before this date?

I think you tried to rollback to 1.7 at this period, right?

31

Yes, as mentioned in the first posts i updated, got some errors referring to the mapping change.
after witing for a while (one or two hours) and no change happened it shut down everything again and deployed the old ES 1.7.1 and started up again, and finding all data lost.

32

So here is what I suspect.

When restarting in 1.7, it was impossible to read newer st files. It might have created empty ones.
That is why we say that you can not downgrade.

I think you are in a bad position here.

I'm not sure if you can do anything than reindexing your data.

33

Ok now the dummy question,

how do i reindex those data?

34

How did you index them the first time?

35

So basically i can downgrade and integrate the data i just need to reindex everything.

36

You can reindex all in Es 2.1 as well if I understand you correctly ...