We follow an es upgrade approach where we reindex the index on every major version upgrade. We noticed during our es9 upgrade from es8, there is a system index .geoip_database which was created in es7 is not allowing es9 to startup. We were redindexing only certain user defined indices on every major version upgrade. Since es9 is not coming up now during upgrade, what are the ways to remediate this now? Even disabling geoip module did not work
from which specific version to which specific version?
Did you review Upgrade Assistant before doing the 8.x --> 9.x upgrade? Did it not flag the indices?
This has been reported a few times, usually just temporarily reverting to the previous 8.x version and dealing with these index issues on 8.x was the route forward.
You made a snapshot before the upgrade, right?
Thanks for the response. We don't use upgrade assistant. We snapshot before upgrade and if the upgrade fails we revert back.
The upgrade assistant exists to detect and help fix issues that may impact the upgrade between major versions, it is the documented and recommended upgrade path.
Without following it there is no guarantee that your upgrade will work without any issues.
Were you able to restore to version 8? If so you should follow the recommended upgrade path and use the upgrade assistant.
Well, in my experience it helps to follow the documentation 
Step 1 is to upgrade to latest 8.19.x patch release - did you do this?
Step 2 is
Run the Upgrade Assistant
The Upgrade Assistant identifies deprecated settings in your configuration and guides you through resolving issues that could prevent a successful upgrade. The Upgrade Assistant also helps resolve issues with older indices created before version 8.0.0, providing the option to reindex older indices or mark them as read-only. To prevent upgrade failures, we strongly recommend you do not skip this step
Often just downgrading the software (say via yum or dnf or apt on linux) to the previously-used release will work, from where you can follow the correct procedure. Of course your snapshots are also a fallback.
Yes it did fortunately.
In our current setup, es upgrade has been happening headless. What i mean is, our webapp on its start up handles reindex logic if it detects a major version change and we update the webapp for new es version changes. There is no separate manual intervention that was needed so far for sometime.
Understood. So I guess you can now open a bug report against your webapp 
And note that a major version upgrade might involve other (documented) changes outside a need to re-index indices created in version N-1 when going from N to N+1. Thats what the Upgrade Assistant is there for, to help you avoid downtime and upgrade successfully. You can "roll your own" methodology obviously, but then you own that risk.
Drive By Help... not sure of the entire upgrade context but you could temporarily disable the geoip, which should clear everything, then upgrade and then reenable. Perhaps that would work... I am surprise when you move the cluster form 7.x to 8.x that there was some leftover that is blocking but perhaps disable / reenable will help in this specific case.
8.x
9.x.
I think it makes sense if upgrade assistant is not used, if I'm not wrong indices in a previous major version would be no problem, but being 2 majors versions behind is the issue.
We dont need geoip module at all. I tried disabling using ingest.geoip.downloader.enabled=false but still es9 tried loading the geoip index created in es7 and aborted the startup.
It looks like an old system index issue. I'd check the startup logs first to confirm .geoip_databases is the blocker. If it's not needed, take a snapshot and remove it so Elasticsearch can recreate it. If you still use GeoIP, verify the supported 7→8→9 upgrade path and share the exact error—it may be a known upgrade compatibility issue.
Yeah, we understood that risk. For us we support only certain number of indices that powers our webapp, during major version upgrade we take care of the necessary changes for all the indices . On the lower environments where we dont have older data from es7 everything was fine and this issue creaped closer up in a production clone environment where we have the data from es7 time. Geoip module was enabled by default and i realise somewhere down that data got loaded. I think we needed the headless upgrade approach because provide our webapp as a hosted solution to customer, and each customer has their own infra for our webapp, and doing a headless upgrade seemed reasonable decision at that point.
Yes, being two version down was the issue, and some how we did not know so far that .geoip index is enabled by default and we have not been managing the index on major version upgrades
If you can't use the upgrade assistant, I would highly recommend at least hitting the /_migration/deprecations API, and not upgrading if that returns any critical deprecations. I believe it would have caught this. The upgrade assistant does other checks, too, so I'd really recommend running it. But at a minimum I'd check the API above.