# Out of memory

**URL:** <https://discuss.elastic.co/t/out-of-memory/88495>\
**Category:** 한국어 질문 및 토론\
**Created:** [June 7, 2017, 3:05am UTC](https://discuss.elastic.co/t/out-of-memory/88495 "2017-06-07T03:05:55Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![TaeSung\_Kim](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@TaeSung\_Kim](https://discuss.elastic.co/u/TaeSung_Kim)\
**Post date:** [June 7, 2017, 3:05am UTC](https://discuss.elastic.co/t/out-of-memory/88495/1 "2017-06-07T03:05:55Z")

</div>

elasticsearch 5.2.2 사용 중입니다.  
갑작스레 노드가 죽어서 확인을 해보니 로그가 남아있지 않았습니다.

우분투에서 journalctl 명령어로 시스템에 남은 로그를 확인해보니 로그는 아래와 같았습니다.

```
Jun 06 12:29:26 hmxelk-03 kernel: python2 invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Jun 06 12:29:26 hmxelk-03 kernel: Out of memory: Kill process 61808 (java) score 649 or sacrifice child**
Jun 06 12:29:26 hmxelk-03 kernel: Killed process 61808 (java) total-vm:384886604kB, anon-rss:8482112kB, file-rss:627280kB
Jun 06 12:29:26 hmxelk-03 kernel: Out of memory: Kill process 1452 (python3) score 2 or sacrifice child
Jun 06 12:29:26 hmxelk-03 kernel: Killed process 1452 (python3) total-vm:230192kB, anon-rss:28524kB, file-rss:3492kB
Jun 06 12:29:26 hmxelk-03 python3[1353]: 2017/06/06 12:29:26.238705 INFO Installed Agent WALinuxAgent-2.2.2 is the most current agent
Jun 06 12:29:26 hmxelk-03 python3[1353]: 2017/06/06 12:29:26.257563 INFO Agent WALinuxAgent-2.2.2 launched with command 'python3 -u /usr/sbin/waagent -run-exthandlers'
Jun 06 12:29:26 hmxelk-03 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Jun 06 12:29:26 hmxelk-03 systemd[1]: elasticsearch.service: Unit entered failed state.
Jun 06 12:29:26 hmxelk-03 systemd[1]: elasticsearch.service: Failed with result 'signal'.

```

MS Azure 가상머신에서 운영 중인데 Out Of Memory가 뜨면서 먼저 python2 가 죽고 이후 java 가 죽고,  
python3 가 죽은 뒤 elasticsearch 노드가 죽었는데 혹시 관련해서 조언을 들을 수 있을지 궁금합니다.

기본적으로 elasticsearch의 설정은 아래와 같습니다.

/etc/default/elasticsearch

```
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited

```

/usr/lib/systemd/elasticsearch.service

```
LimitMEMLOCK=infinity

```

/etc/security/limits.conf

```
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

```

/etc/elasticsearch/elasticsearch.yml

```
bootstrap.memory_lock: true
indices.fielddata.cache.size: 75%
indices.memory.index_buffer_size: 10%

```

GET \_cluster/settings

```
    "indices": {
      "breaker": {
        "fielddata": {
          "limit": "77%"
        },
        "request": {
          "limit": "50%"
        },
        "total": {
          "limit": "80%"
        }
      }
    }
  }

```

경험 있으신 분들이 계시다면 조언 부탁드립니다. 🙂  
감사합니다.

---

<div class="post-metadata">

**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\
**Post date:** [June 7, 2017, 3:55am UTC](https://discuss.elastic.co/t/out-of-memory/88495/2 "2017-06-07T03:55:00Z")

</div>

> [@TaeSung\_Kim](#):
>
> Killed process 61808 (java) total-vm:384886604kB, anon-rss:8482112kB, file-rss:627280kB

oom killer 작동 시점에 ES가 물리램을 8.6GB 정도 사용 중이었군요. Azure 장비의 물리램을 몇 기가이고, ES의 Heap size는 몇 GB를 설정하셨는지요?

---

<div class="post-metadata">

**Author:** ![TaeSung\_Kim](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@TaeSung\_Kim](https://discuss.elastic.co/u/TaeSung_Kim)\
**Post date:** [June 7, 2017, 3:59am UTC](https://discuss.elastic.co/t/out-of-memory/88495/3 "2017-06-07T03:59:14Z")

</div>

답변 주셔서 감사합니다.

TOTAL : 14GB  
ES : 7GB

설정했습니다.

---

<div class="post-metadata">

**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\
**Post date:** [June 7, 2017, 4:20am UTC](https://discuss.elastic.co/t/out-of-memory/88495/4 "2017-06-07T04:20:52Z")

</div>

예. ES dedicated된 node라서 전체 RAM의 50%를 ES Heap에 할당하신 것 같군요.

python이랑 ES 다 합쳐도 10GB는 안 되는데, (페북 그룹에 다른 분이 댓글에서 언급하신 커널 버그가 아니라면) 어떤 process가 나머지 메모리를 사용 중이었는지 찾아보셔야겠군요.

큰 도움이 안 되서 죄송하네요 ^^;

---

<div class="post-metadata">

**Author:** ![TaeSung\_Kim](https://avatars.discourse-cdn.com/v4/letter/t/c2a13f/32.png) [@TaeSung\_Kim](https://discuss.elastic.co/u/TaeSung_Kim)\
**Post date:** [June 7, 2017, 5:35am UTC](https://discuss.elastic.co/t/out-of-memory/88495/5 "2017-06-07T05:35:56Z")

</div>

도움이 안되다니요.  
답변 주셔서 감사하죠.

글자수 제한 때문에 full log를 다 올린 게 아니라서요.  
그래도 많이 배웠습니다.  
감사합니다.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 5:36am UTC](https://discuss.elastic.co/t/out-of-memory/88495/6 "2017-07-05T05:36:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.