# Pytorch\_inference silenty disappear during reindex using pretrained machine learning model **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning, docker\ **Created:** [April 27, 2023, 4:35am UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896 "2023-04-27T04:35:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![tomotaka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomotaka/32/120298_2.png) [@tomotaka](https://discuss.elastic.co/u/tomotaka)\ **Post date:** [April 27, 2023, 4:35am UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896/1 "2023-04-27T04:35:51Z")
We are planning to build a vector-based search application with pretrained machine learning model which is based on mBERT model. So now I wrote some code to check how Elasticsearch works and I found that pytorch\_inference process randomly disappear during vector computation step (reindex). There is no stacktrace or Elasticsearch WARN/ERROR logs about it. Our program keeps polling to check reindex task finished or not, the progress shows "created" number never increases after pytorch\_inference process had gone. The timing of pytorch\_inference disappear seems very random, sometimes after computed 1000 documents, sometimes after computed 48000... I run Elasticsearch on docker container (docker-compose) and I gave 12 cores to make Elasticsearch compute ML inference faster. Any suggestion? Thanks. ---
**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\ **Post date:** [April 27, 2023, 10:41am UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896/2 "2023-04-27T10:41:09Z")
Hi @tomotaka The **pytorch\_inference** process is where the model is evaluated so if that has disappeared the reindex task cannot progress. I am very surprised there is nothing in the logs, try searching the logs for the model Id of your model. Depending on the cause of the failure the GET \_stats endpoint may report something interesting. ```auto GET _ml/trained_models//_stats ``` Look for the fields `deployment_stats. allocation_status` and `deployment_stats.nodes.routing_state`, what is the status? I will try to recreate the problem of pytorch\_inference disappearing. What version of Elasticsearch are you using and are you using the Elasticsearch docker image from [https://www.docker.elastic.co/](https://www.docker.elastic.co/) or did you create it yourself? ---
**Author:** ![tomotaka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomotaka/32/120298_2.png) [@tomotaka](https://discuss.elastic.co/u/tomotaka)\ **Post date:** [April 27, 2023, 3:25pm UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896/3 "2023-04-27T15:25:23Z")
Hi @dkyle , thank you for the reply. > The **pytorch\_inference** process is where the model is evaluated so if that has disappeared the reindex task cannot progress. I am very surprised there is nothing in the logs, try searching the logs for the model Id of your model. I search about the model Id and machine learning related things from container stdout output and looked for in /var/log/ and /usr/share/elasitcsaerch inside the container but I could not find Elasticserach log file. Does Elasticsearch docker image have not only stdout but also logfile output? > Look for the fields `deployment_stats. allocation_status` and `deployment_stats.nodes.routing_state`, what is the status? OK, I will try during my work time tomorrow (I live in Japan so it is going to be night now...) > I will try to recreate the problem of pytorch\_inference disappearing. What version of Elasticsearch are you using and are you using the Elasticsearch docker image from [https://www.docker.elastic.co/](https://www.docker.elastic.co/) or did you create it yourself? Here is our Dockerfile. It is based on elasticsearch 8.7.0 official image and I think there's not so big addition. ```auto FROM docker.elastic.co/elasticsearch/elasticsearch:8.7.0 # set the specific password for Elasticsearch ENV ELASTIC_PASSWORD=elastic RUN bin/elasticsearch-plugin install analysis-kuromoji RUN bin/elasticsearch-plugin install analysis-icu RUN sysctl -w vm.max_map_count=262144 ``` Thank you for your effort! ---
**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\ **Post date:** [April 28, 2023, 9:14am UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896/4 "2023-04-28T09:14:56Z")
I noticed the docker file is setting `vm.max_map_count`. This should be done on the host machine not the docker vm. > [@tomotaka](#): > > `RUN sysctl -w vm.max_map_count=262144` Here are Elasticsearch's instructions for setting `vm.max_map_count`. > **[Install Elasticsearch with Docker | Elasticsearch Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/docker.html#_set_vm_max_map_count_to_at_least_262144)** ---
**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\ **Post date:** [May 24, 2023, 8:05am UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896/5 "2023-05-24T08:05:57Z")
Hi @tomotaka It is quite likely that the pytorch\_inference process is being terminated by the Out Of Memory Killer (the process runs at lower priority so the OOM killer will choose to terminated pytorch\_inference rather than Elasticsearch) Can you run the container with more memory? Note this the not the same as the JVM heap memory used by Elasticsearch, there must be memory in the container outside of the JWM for the pytorch\_inference process. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 21, 2023, 8:06am UTC](https://discuss.elastic.co/t/pytorch-inference-silenty-disappear-during-reindex-using-pretrained-machine-learning-model/330896/6 "2023-06-21T08:06:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.