Bulk process stuck

Elastic Stack Elasticsearch
1

when bulk document ,Bulk operation stuck in running state.

What may be the reason?

elasticsearch:6.4.2
3 nodes

  •   indices:data/write/bulk                      CpJ4Sde5TXqvPG_UKFSRZQ:233020 -                            transport 1556587908016 01:31:48 1.6h        172.16.217.20 CpJ4Sde

  • indices:data/write/bulk[s]                   CpJ4Sde5TXqvPG_UKFSRZQ:233021 CpJ4Sde5TXqvPG_UKFSRZQ:233020 transport 1556587908020 01:31:48 1.6h        172.16.217.20 CpJ4Sde

  • indices:data/write/bulk[s]                   _5PM_TvgSvmoxZFFl3D1Lw:215192 CpJ4Sde5TXqvPG_UKFSRZQ:233021 netty     1556587933113 01:32:13 1.6h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk[s][p]                _5PM_TvgSvmoxZFFl3D1Lw:215193 _5PM_TvgSvmoxZFFl3D1Lw:215192 direct    1556587933113 01:32:13 1.6h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk                      CpJ4Sde5TXqvPG_UKFSRZQ:233152 -                             transport 1556587971902 01:32:51 1.6h        172.16.217.20 CpJ4Sde

  • indices:data/write/bulk[s]                   CpJ4Sde5TXqvPG_UKFSRZQ:233153 CpJ4Sde5TXqvPG_UKFSRZQ:233152 transport 1556587971910 01:32:51 1.6h        172.16.217.20 CpJ4Sde

  • indices:data/write/bulk[s]                   _5PM_TvgSvmoxZFFl3D1Lw:215316 CpJ4Sde5TXqvPG_UKFSRZQ:233153 netty     1556587991923 01:33:11 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk[s][p]                _5PM_TvgSvmoxZFFl3D1Lw:215317 _5PM_TvgSvmoxZFFl3D1Lw:215316 direct    1556587991924 01:33:11 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk                      _5PM_TvgSvmoxZFFl3D1Lw:215366 -                             transport 1556588010842 01:33:30 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk[s]                   _5PM_TvgSvmoxZFFl3D1Lw:215367 _5PM_TvgSvmoxZFFl3D1Lw:215366 transport 1556588010848 01:33:30 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk[s][p]                _5PM_TvgSvmoxZFFl3D1Lw:215369 _5PM_TvgSvmoxZFFl3D1Lw:215367 direct    1556588010848 01:33:30 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk                      _5PM_TvgSvmoxZFFl3D1Lw:215432 -                             transport 1556588037252 01:33:57 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk[s]                   _5PM_TvgSvmoxZFFl3D1Lw:215434 _5PM_TvgSvmoxZFFl3D1Lw:215432 transport 1556588037254 01:33:57 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk                      _5PM_TvgSvmoxZFFl3D1Lw:215499 -                             transport 1556588067444 01:34:27 1.5h        172.16.215.15 _5PM_Tv

  • indices:data/write/bulk[s]                   _5PM_TvgSvmoxZFFl3D1Lw:215500 _5PM_TvgSvmoxZFFl3D1Lw:215499 transport 1556588067448 01:34:27 1.5h        172.16.215.15 _5PM_Tv

details info

"i6Idpt6oSb-_T8GhBp5DFg:222210" : {
  "node" : "i6Idpt6oSb-_T8GhBp5DFg",
  "id" : 222210,
  "type" : "transport",
  "action" : "indices:data/write/bulk",
  "description" : "requests[2076], indices[logs-39ae8b6acd104b1e823332cd6f64d415-2019-04-30, logs-78e238e645b84b50ada2b3ea45e232a7-2019-04-30]",
  "start_time" : "2019-04-30T02:05:01.364Z",
  "start_time_in_millis" : 1556589901364,
  "running_time" : "1h",
  "running_time_in_nanos" : 3888982728848,
  "cancellable" : false,
  "headers" : { },
  "children" : [
    {
      "node" : "i6Idpt6oSb-_T8GhBp5DFg",
      "id" : 222211,
      "type" : "transport",
     "action" : "indices:data/write/bulk[s]",
      "status" : {
        "phase" : "rerouted"
      },
      "description" : "requests[2009], index[logs-78e238e645b84b50ada2b3ea45e232a7-2019-04-30]",
      "start_time" : "2019-04-30T02:05:01.366Z",
      "start_time_in_millis" : 1556589901366,
      "running_time" : "1h",
      "running_time_in_nanos" : 3888980969875,
      "cancellable" : false,
      "parent_task_id" : "i6Idpt6oSb-_T8GhBp5DFg:222210",
      "headers" : { },
      "children" : [
        {
          "node" : "_5PM_TvgSvmoxZFFl3D1Lw",
          "id" : 219793,
          "type" : "netty",
          "action" : "indices:data/write/bulk[s]",
          "status" : {
            "phase" : "waiting_on_primary"
          },
          "description" : "requests[2009], index[logs-78e238e645b84b50ada2b3ea45e232a7-2019-04-30]",
          "start_time" : "2019-04-30T02:06:15.118Z",
          "start_time_in_millis" : 1556589975118,
          "running_time" : "1h",
          "running_time_in_nanos" : 3815228269643,
          "cancellable" : false,
          "parent_task_id" : "i6Idpt6oSb-_T8GhBp5DFg:222211",
          "headers" : { },
          "children" : [
            {
              "node" : "_5PM_TvgSvmoxZFFl3D1Lw",
              "id" : 219794,
              "type" : "direct",
              "action" : "indices:data/write/bulk[s][p]",
              "status" : {
                "phase" : "primary"
              },
              "description" : "requests[2009], index[logs-78e238e645b84b50ada2b3ea45e232a7-2019-04-30]",
              "start_time" : "2019-04-30T02:06:15.119Z",
              "start_time_in_millis" : 1556589975119,
              "running_time" : "1h",
              "running_time_in_nanos" : 3815227909921,
              "cancellable" : false,
              "parent_task_id" : "_5PM_TvgSvmoxZFFl3D1Lw:219793",
              "headers" : { }
            }
          ]
        }
      ]
    }
  ]
},
2

How large is your bulk request?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.