Elasticsearch stopped working , it is not extracting contents from documents

priyankaa · October 16, 2023, 7:24am

due to disk storage got full , Elasticsearch was stopped working , so we have now increased it , still after increasing disk storage Elasticsearch is not working , I performed reindexing as per my senior suggestion , but got no success , what can be possible causes for Elasticsearch not working , How can I get to know its actual cause for not working ?

dityudha · October 16, 2023, 8:18am

can you send the log error message? default config located on /var/log/elasticsearch

are you already tried restart the elasticserach service?

priyankaa · October 16, 2023, 9:57am

@dityudha
I am new to elasticsearch , how can I check error log message?
I tried restarting elasticsearch but same elasticsearch failing to extract contents from documents.

PodarcisMuralis · October 16, 2023, 6:14pm

Typically logs are under elasticserarch/logs.
If you are on linux try

tail -f logname.log
Or
cat logname.log

You can restar elasticsearch with
sudo systemctl restart elasticsearch if you have elasticsearch service.

priyankaa · October 17, 2023, 2:10pm

how can I check log on windows? is it possible using cmd?

dadoonet · October 17, 2023, 3:41pm

The documentation tells you where to find the logs. See Install Elasticsearch with .zip on Windows | Elasticsearch Guide [8.10] | Elastic at the bottom of the page.

priyankaa · October 17, 2023, 3:58pm

[2023-10-16T01:29:59,141][INFO ][o.e.x.m.MlDailyMaintenanceService] [EC2AMAZ-I4482TI] triggering scheduled [ML] maintenance tasks
[2023-10-16T01:29:59,141][INFO ][o.e.x.s.SnapshotRetentionTask] [EC2AMAZ-I4482TI] starting SLM retention snapshot cleanup task
[2023-10-16T01:29:59,350][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [EC2AMAZ-I4482TI] Deleting expired data
[2023-10-16T01:29:59,351][INFO ][o.e.x.s.SnapshotRetentionTask] [EC2AMAZ-I4482TI] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2023-10-16T01:29:59,406][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [EC2AMAZ-I4482TI] Successfully deleted [0] unused stats documents
[2023-10-16T01:29:59,485][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [EC2AMAZ-I4482TI] Completed deletion of expired ML data
[2023-10-16T01:29:59,500][INFO ][o.e.x.m.MlDailyMaintenanceService] [EC2AMAZ-I4482TI] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2023-10-16T17:07:04,943][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip databases
[2023-10-16T17:07:04,943][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2023-10-16T17:07:05,130][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip database [GeoLite2-ASN.mmdb]
[2023-10-16T17:07:05,848][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] downloading geoip database [GeoLite2-ASN.mmdb] to [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-ASN.mmdb.tmp.gz]
[2023-10-16T17:07:05,864][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updated geoip database [GeoLite2-ASN.mmdb]
[2023-10-16T17:07:05,864][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip database [GeoLite2-City.mmdb]
[2023-10-16T17:07:06,020][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [EC2AMAZ-I4482TI] evicted [0] entries from cache after reloading database [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-ASN.mmdb]
[2023-10-16T17:07:06,036][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] successfully reloaded changed geoip database file [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-ASN.mmdb]
[2023-10-16T17:07:08,579][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] downloading geoip database [GeoLite2-City.mmdb] to [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-City.mmdb.tmp.gz]
[2023-10-16T17:07:08,595][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updated geoip database [GeoLite2-City.mmdb]
[2023-10-16T17:07:08,595][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip database [GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,224][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] downloading geoip database [GeoLite2-Country.mmdb] to [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-Country.mmdb.tmp.gz]
[2023-10-16T17:07:09,302][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updated geoip database [GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,380][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [EC2AMAZ-I4482TI] evicted [0] entries from cache after reloading database [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,380][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] successfully reloaded changed geoip database file [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,568][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [EC2AMAZ-I4482TI] evicted [0] entries from cache after reloading database [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-City.mmdb]
[2023-10-16T17:07:09,568][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] successfully reloaded changed geoip database file [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-City.mmdb]

this are contents I got from log file , can you please go through it once @dadoonet sir?

dadoonet · October 17, 2023, 4:12pm

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

And please share the full logs from the start.

priyankaa · October 18, 2023, 5:47am

[2023-10-16T01:29:59,141][INFO ][o.e.x.m.MlDailyMaintenanceService] [EC2AMAZ-I4482TI] triggering scheduled [ML] maintenance tasks
[2023-10-16T01:29:59,141][INFO ][o.e.x.s.SnapshotRetentionTask] [EC2AMAZ-I4482TI] starting SLM retention snapshot cleanup task
[2023-10-16T01:29:59,350][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [EC2AMAZ-I4482TI] Deleting expired data
[2023-10-16T01:29:59,351][INFO ][o.e.x.s.SnapshotRetentionTask] [EC2AMAZ-I4482TI] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2023-10-16T01:29:59,406][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [EC2AMAZ-I4482TI] Successfully deleted [0] unused stats documents
[2023-10-16T01:29:59,485][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [EC2AMAZ-I4482TI] Completed deletion of expired ML data
[2023-10-16T01:29:59,500][INFO ][o.e.x.m.MlDailyMaintenanceService] [EC2AMAZ-I4482TI] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2023-10-16T17:07:04,943][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip databases
[2023-10-16T17:07:04,943][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2023-10-16T17:07:05,130][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip database [GeoLite2-ASN.mmdb]
[2023-10-16T17:07:05,848][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] downloading geoip database [GeoLite2-ASN.mmdb] to [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-ASN.mmdb.tmp.gz]
[2023-10-16T17:07:05,864][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updated geoip database [GeoLite2-ASN.mmdb]
[2023-10-16T17:07:05,864][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip database [GeoLite2-City.mmdb]
[2023-10-16T17:07:06,020][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [EC2AMAZ-I4482TI] evicted [0] entries from cache after reloading database [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-ASN.mmdb]
[2023-10-16T17:07:06,036][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] successfully reloaded changed geoip database file [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-ASN.mmdb]
[2023-10-16T17:07:08,579][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] downloading geoip database [GeoLite2-City.mmdb] to [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-City.mmdb.tmp.gz]
[2023-10-16T17:07:08,595][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updated geoip database [GeoLite2-City.mmdb]
[2023-10-16T17:07:08,595][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updating geoip database [GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,224][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] downloading geoip database [GeoLite2-Country.mmdb] to [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-Country.mmdb.tmp.gz]
[2023-10-16T17:07:09,302][INFO ][o.e.i.g.GeoIpDownloader ] [EC2AMAZ-I4482TI] updated geoip database [GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,380][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [EC2AMAZ-I4482TI] evicted [0] entries from cache after reloading database [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,380][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] successfully reloaded changed geoip database file [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-Country.mmdb]
[2023-10-16T17:07:09,568][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [EC2AMAZ-I4482TI] evicted [0] entries from cache after reloading database [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-City.mmdb]
[2023-10-16T17:07:09,568][INFO ][o.e.i.g.DatabaseNodeService] [EC2AMAZ-I4482TI] successfully reloaded changed geoip database file [C:\Users\ADMINI~1\AppData\Local\Temp\2\elasticsearch\geoip-databases\e1pFucaYQx6MpQjXYdBBGw\GeoLite2-City.mmdb]

Thanks for your suggestion @dadoonet sir , please check now sir

dadoonet · October 18, 2023, 7:09am

All is good and working from what I can see from this part of the logs as you did not do that:

And please share the full logs from the start.

So what is your problem? Could you describe what you are doing and what does not work?

priyankaa · October 18, 2023, 8:07am

Basically before some days we faced disk storage full issue on my remote desktop, so we increased it but that time elasticsearch stopped scanning documents

after incresing disk space one of my senior suggest me to perform reindexinig , so I did it considering it will help me to link old and new indexes but after successfull reindexing when I scan one document , it didn't got parsed , and I didnt got any content of that document

now my main concern is to bring back elasticsearch in working without loosing old documents linking + it should scan new documents too.

please help me in this matter , how can I achieve this.

also do you want all logs from starting ? please let me know sir.

dadoonet · October 18, 2023, 12:00pm

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

I guess that the screenshot is from that period, right?

How did you do that?

Could you share an example of what you are saying? I don't understand what you are meaning by "it didn't got parsed". Is the document "empty"? Or does the document not exist at all?

Where are the old documents? This is a similar question to "How did you do that?" I just asked...

What do you mean by "scan"? I don't understand.

Share your architecture.

priyankaa · October 18, 2023, 12:46pm

1.error was high disk watermark [90%] exceeded on C:\elasticsearch-7.17.0\data\nodes\0 , so we increased C drive space.Yes screenshot is from that period.

I performed reindexing as follows

GET resumes/_search
PUT temp_index

POST _reindex
{
  "source": {
    "index": "resumes"
  },
  "dest": {
    "index": "temp_index"
  }
}

GET temp_index/_search

DELETE resumes

PUT resumes

POST _reindex
{
  "source": {
    "index": "temp_index"
  },
  "dest": {
    "index": "resumes"
  }
}

3."it didn't got parsed". means In my application where I am using elasticsearch when I upload one resume document it should give me contents from that resume document
ex : I am uploading resume document so in backend I have added code in such manner it will return email id , mobile no , zipcode , name of person and all other text from that document. right now when I am uploading resume document it is returning no contents .

4.there are 33+ thousands resume documents that already uploaded and we have their contents in database so I dont want to loose this old documents indexing

- it should scan new documents too. means after reindexing , when I upload another resume document , I am expecting Elasticsearch will return me its contents.

dadoonet · October 18, 2023, 1:16pm

Well... Where is the mapping? If you did not set the right mapping, I doubt this will work out of the box.

Ok but this is something in your code then. Elasticsearch does not extract such data as far as I know. So how you are doing this?

Best thing to do is to start doing a backup (snapshot).

Yes. I'm expecting that if your application is sending a document to be indexed by Elasticsearch, if it is successfully indexed, and that the index has been refreshed, you should see your document.
First thing:

check the mapping
check the response from elasticsearch when you are indexing

priyankaa · October 18, 2023, 1:17pm

ok , I will check mapping and get back to you.

priyankaa · October 21, 2023, 5:16pm

@dadoonet sir I guess somethisng wrong happed with fscrawler

dadoonet · October 21, 2023, 10:24pm

I guess it's "ok".

But you should not be running 7.17.0. There's a more recent version of 7.17.

system · November 18, 2023, 10:25pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.