# Failed action with response of 400, dropping action

**URL:** <https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802>\
**Category:** Logstash\
**Created:** [September 8, 2015, 1:29am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802 "2015-09-08T01:29:46Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![angelsmile](https://avatars.discourse-cdn.com/v4/letter/a/e19b73/32.png) [@angelsmile](https://discuss.elastic.co/u/angelsmile)\
**Post date:** [September 8, 2015, 1:29am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/1 "2015-09-08T01:29:46Z")

</div>

Hi,

For the following log:

```
Job Start Summary: jobid=2291646.r-man2 project=p08 user=rnt651@raijin1 queue=express-node ncpus=16 mem_requested=8gb walltime=16:00:00 cpu_hrs=768.0 jobfs_local=8gb jobfs_global=None

```

When I trim ':' with logstash kv filter, the log can be passed to ES, but then the walltime becomes '160000'.

```
kv {
   trim => "<>:"
   field_split => " "
   prefix => "job_"
}

```

If I don't trim ':' I get the following error and the log is dropped by ES

{:timestamp=\>"2015-09-07T17:33:30.017000+1000", :message=\>"failed action with response of 400, dropping action: ["index", {:\_id=\>nil, :\_index=\>"logstash-2015.09.07", :\_type=\>"program", :\_routing=\>nil}, #0}, @accessors=#"r-man2", "message"=\>"Job Start Summary: jobid=2291646.r-man2 project=p08 user=rnt651@raijin1 queue=express-node ncpus=16 mem\_requested=8gb walltime=16:00:00 cpu\_hrs=768.0 jobfs\_local=8gb jobfs\_global=None", "offset"=\>253322067, "path"=\>"/var/spool/PBS/server\_logs/20150907", "program"=\>"pbs\_server", "type"=\>"program", "@version"=\>"1", "@timestamp"=\>"2015-09-07T07:33:08.000Z", "pbs\_timestamp"=\>"09/07/2015 17:33:08", "pbs\_event\_class"=\>"job", "pbs\_node"=\>"Server@r-man2", "pbs\_object\_type"=\>"job", "job\_id"=\>"2291646.r-man2", "received\_from"=\>"r-man2", "pbs\_host"=\>"r-man2", "pbs\_program"=\>"pbs\_server", "received\_at"=\>"09/07/2015 17:33:08", "tags"=\>["pbs\_production", "pbs\_job"], "job\_jobid"=\>"2291646.r-man2", "job\_project"=\>"p08", "job\_user"=\>"rnt651@raijin1", "job\_queue"=\>"express-node", "job\_ncpus"=\>16, "job\_mem\_requested"=\>"8gb", "job\_walltime"=\>"16:00:00", "job\_cpu\_hrs"=\>768.0, "job\_jobfs\_local"=\>"8gb", "job\_jobfs\_global"=\>"None", "pbs\_user"=\>"rnt651@raijin1"}, @lut={"[pbs\_object\_type]"=\>[{"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2291646.r-man2 project=p08 user=rnt651@raijin1 queue=express-node ncpus=16 mem\_requested=8gb walltime=16:00:00 cpu\_hrs=768.0 jobfs\_local=8gb jobfs\_global=None", "offset"=\>253322067, "path"=\>"/var/spool/PBS/server\_logs/20150907", "program"=\>"pbs\_server", "type"=\>"program", "@version"=\>"1", "@timestamp"=\>"2015-09-07T07:33:08.000Z", "pbs\_timestamp"=\>"09/07/2015 17:33:08", "pbs\_event\_class"=\>"job", "pbs\_node"=\>"Server@r-man2", "pbs\_object\_type"=\>"job", "job\_id"=\>"2291646.r-man2", "received\_from"=\>"r-man2", "pbs\_host"=\>"r-man2", "pbs\_program"=\>"pbs\_server", "received\_at"=\>"09/07/2015 17:33:08", "tags"=\>["pbs\_production", "pbs\_job"], "job\_jobid"=\>"2291646.r-man2", "job\_project"=\>"p08", "job\_user"=\>"rnt651@raijin1", "job\_queue"=\>"express-node", "job\_ncpus"=\>16, "job\_mem\_requested"=\>"8gb", "job\_walltime"=\>"16:00:00", "job\_cpu\_hrs"=\>768.0, "job\_jobfs\_local"=\>"8gb", "job\_jobfs\_global"=\>"None", "pbs\_user"=\>"rnt651@raijin1"}, "pbs\_object\_type"], "pbs\_object"=\>[{"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2291646.r-man2 project=p08 user=rnt651@raijin1 queue=express-node ncpus=16 mem\_requested=8gb walltime=16:00:00 cpu\_hrs=768.0 jobfs\_local=8gb jobfs\_global=None", "offset"=\>253322067, "path"=\>"/var/spool/PBS/server\_logs/20150907", "program"=\>"pbs\_server", "type"=\>"program", "@version"=\>"1", "@timestamp"=\>"2015-09-07T07:33:08.000Z", "pbs\_timestamp"=\>"09/07/2015 17:33:08", "pbs\_event\_class"=\>"job", "pbs\_node"=\>"Server@r-man2", "pbs\_object\_type"=\>"job", "job\_id"=\>"2291646.r-man2", "received\_from"=\>"r-man2", "pbs\_host"=\>"r-man2", "pbs\_program"=\>"pbs\_server", "received\_at"=\>"09/07/2015 17:33:08", "tags"=\>["pbs\_production", "pbs\_job"], "job\_jobid"=\>"2291646.r-man2", "job\_project"=\>"p08", "job\_user"=\>"rnt651@raijin1", "job\_queue"=\>"express-node", "job\_ncpus"=\>16, "job\_mem\_requested"=\>"8gb", "job\_walltime"=\>"16:00:00", "job\_cpu\_hrs"=\>768.0, "job\_jobfs\_local"=\>"8gb", "job\_jobfs\_global"=\>"None", "pbs\_user"=\>"rnt651@raijin1"}, "pbs\_object"], "tags"=\>[{"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2291646.r-man2 project=p08 user=rnt651@raijin1 queue=express-node ncpus=16 mem\_requested=8gb walltime=16:00:00 cpu\_hrs=768.0 jobfs\_local=8gb jobfs\_global=None",  
...........  
@metadata\_accessors=#0}, @lut={}\>, @cancelled=false\>]", :level=\>:warn}

**Any idea why the log is dropped? I'm running logstash 1.5.4 and ES 1.4.5 on a CentOS 6.6 server.**

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 8, 2015, 1:43am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/2 "2015-09-08T01:43:26Z")

</div>

Try looking in the ES logs to see if there is something that corresponds.

(Also, upgrade ES 😉 )

---

<div class="post-metadata">

**Author:** ![angelsmile](https://avatars.discourse-cdn.com/v4/letter/a/e19b73/32.png) [@angelsmile](https://discuss.elastic.co/u/angelsmile)\
**Post date:** [September 8, 2015, 1:48am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/3 "2015-09-08T01:48:44Z")

</div>

Many thanks for your quick response Mark. I've confirmed there is nothing corresponding to this in ES log. I will upgrade ES to see if it can be resolved.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 8, 2015, 7:21am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/4 "2015-09-08T07:21:58Z")

</div>

It looks like only the part after "Job Start Summary: " is a proper k/v list. I would therefore suggest trying to first use a grok filter to isolate the k/v list and then applying the kv filter on the field containing the list without the trim parameter specified.

---

<div class="post-metadata">

**Author:** ![angelsmile](https://avatars.discourse-cdn.com/v4/letter/a/e19b73/32.png) [@angelsmile](https://discuss.elastic.co/u/angelsmile)\
**Post date:** [September 9, 2015, 6:49am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/5 "2015-09-09T06:49:07Z")

</div>

I have upgraded ES to v1.7.1 but this problem still happens...

---

<div class="post-metadata">

**Author:** ![angelsmile](https://avatars.discourse-cdn.com/v4/letter/a/e19b73/32.png) [@angelsmile](https://discuss.elastic.co/u/angelsmile)\
**Post date:** [September 9, 2015, 6:57am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/7 "2015-09-09T06:57:52Z")

</div>

Hi Christian,

Many thanks for your suggestion. I have followed your advice but unfortunately I still get the same error.

```
grok {
     match => ["message", "Job Start Summary: %{GREEDYDATA:kv_target}"]
 }
kv {
    source => "kv_target"
    field_split => " "
   prefix => "job_"
}

```

{:timestamp=\>"2015-09-09T16:46:47.212000+1000", :message=\>"failed action with response of 400, dropping action: ["index", {:id=\>nil, :index=\>"logstash-2015.09.09", :type=\>"program", :routing=\>nil}, #@metadata={"retry\_count"=\>0}, @accessors=#@store={"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2  
325253.r-man2 project=v71 user=mxz561@raijin3 queue=express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "offset"=\>182499027, "path"=\>"/var/spool/PBS/server\_logs/20150909", "program"=\>"pbs\_server", "type"=\>"program", "@version"=\>"1", "@timestamp"=\>"2015-09-09T06:46:05.000Z", "pbs\_timestamp"=\>"09/09/2015 16:46:05", "pbs\_event\_class"=\>"job", "pbs\_node"=\>"Server@r-man2", "pbs\_object\_type"=\>"job", "job\_id"=\>"2325253.r-man2", "received\_from"=\>"r-man2", "pbs\_host"=\>"r-man2", "pbs\_program"=\>"pbs\_server", "received\_at"=\>"09/09/2015 16:46:05", "tags"=\>["pbs\_production", "pbs\_job"], "kv\_target"=\>"jobid=2325253.r-man2 project=v71 user=mxz561@raijin3 queue=  
express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "job\_jobid"=\>"2325253.r-man2", "job\_project"=\>"v71", "job\_user"=\>"mxz561@raijin3", "job\_queue"=\>"express-node", "job\_ncpus"=\>16,  
"job\_mem\_requested"=\>"16gb", "job\_walltime"=\>"24:00:00", "job\_cpu\_hrs"=\>1152.0, "job\_jobfs\_local"=\>"100mb", "job\_jobfs\_global"=\>"None", "pbs\_user"=\>"mxz561@raijin3"}, @lut={"[pbs\_object\_type]"=\>[{"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2325253.r-man2 project=v71 user=mxz561@raijin3 queue=express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "offset"=\>182499027, "path"=\>"/var/spool/PBS/server\_logs/20150909", "program"=\>"pbs\_server", "type"=\>"program", "@version"=\>"1", "@timestamp"=\>"2015-09-09T06:46:05.000Z", "pbs\_timestamp"=\>"09/09/2015 16:46:05", "pbs\_event\_class"=\>"job", "pbs\_node"=\>"Server@r-man2", "pbs\_object\_type"=\>"job", "job\_id"=\>"2325253.r-man2", "received\_from"=\>"r-man2", "pbs\_host"=\>"r-man2", "pbs\_program"=\>"pbs\_server", "received at"=\>"09/09/2015 16:46:05", "tags"=\>["pbsproduction", "pbs\_job"], "kv\_target"=\>"jobid=2325253.r-man2 project=v71 user=mxz561@raijin3 queue=express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "job\_jobid"=\>"2325253.r-man2", "job\_project"=\>"v71", "job\_user"=\>"mxz561@raijin3", "job\_queue"=\>"express-node", "job\_ncpus"=\>16, "job\_mem\_requested"=\>"16gb", "job\_walltime"=\>"24:00:00", "job\_cpu\_hrs"=\>1152.0, "job\_jobfs\_local"=\>"100mb", "job\_jobfs\_global"=\>"None", "pbs\_user"=\>"mxz561@raijin3"}, "pbs\_object\_type"], "pbs\_object"=\>[{"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2325253.r-man2 project=v71 user=mxz561@raijin3 queue=express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "offset"=\>182499027, "path"=\>"/var/  
spool/PBS/server\_logs/20150909", "program"=\>"pbs\_server", "type"=\>"program", "@version"=\>"1", "@timestamp"=\>"2015-09-09T06:46:05.000Z", "pbs\_timestamp"=\>"09/09/2015 16:46:05", "pbs\_event\_class"=\>"job", "pbs\_node"=\>"Server@r-man2", "pbs\_object\_type"=\>"job", "job\_id"=\>"2325253.r-man2", "received\_from"=\>"r-man2", "pbs\_host"=\>"r-man2", "pbs\_progr  
am"=\>"pbs\_server", "received\_at"=\>"09/09/2015 16:46:05", "tags"=\>["pbs\_production", "pbs\_job"], "kv\_target"=\>"jobid=2325253.r-man2 project=v71 user=mxz561@raijin3 queue=express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "job\_jobid"=\>"2325253.r-man2", "job\_project"=\>"v71", "job\_user"=\>"mxz561@raijin3  
", "job\_queue"=\>"express-node", "job\_ncpus"=\>16, "job\_mem\_requested"=\>"16gb", "job\_walltime"=\>"24:00:00", "job\_cpu\_hrs"=\>1152.0, "job\_jobfs\_local"=\>"100mb", "job\_jobfs\_global"=\>"None", "pbs\_user"=\>"mxz561@raijin3"}, "pbs\_object"],  
"tags"=\>[{"host"=\>"r-man2", "message"=\>"Job Start Summary: jobid=2325253.r-man2 project=v71 user=mxz561@raijin3 queue=express-node ncpus=16 mem\_requested=16gb walltime=24:00:00 cpu\_hrs=1152.0 jobfs\_local=100mb jobfs\_global=None", "offset"=\>182499027, "path"=\>"/var/spool/PBS/server\_logs/20150909",

---

<div class="post-metadata">

**Author:** ![angelsmile](https://avatars.discourse-cdn.com/v4/letter/a/e19b73/32.png) [@angelsmile](https://discuss.elastic.co/u/angelsmile)\
**Post date:** [September 11, 2015, 5:08am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/8 "2015-09-11T05:08:20Z")

</div>

I changed the prefix from "job\_" to something else and the logs pass ES now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:29am UTC](https://discuss.elastic.co/t/failed-action-with-response-of-400-dropping-action/28802/9 "2017-07-06T05:29:27Z")

</div>