Hi,
I am seeing quite a few warnings in my logstash saying "Failed action with response of 400, dropping action". In elasticsearch log I am seeing corresponding error as failed to execute bulk item (index) index. This is not happening every time. Can some one let me know what is going wrong.
I have a simple ES Cluster with 3 ES Data-Master node & 1 ES Client Node. The index is having 7 shards with replication factor 1. All machines are having 64 GB RAM, 32 Core, 2TB Disk & running Debian 8.
A single logstash is pushing logs to ES Client with the below configurations:-
output {
elasticsearch {
index => "jab-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
cluster => "elasticsearch"
host => "es-client"
port => "9300"
protocol => "transport"
}
file {
path => "/var/log/shop/%{env}/%{app}/%{app}_%{host}_%{[@metadata][app_log_time]}.log"
}
}
The ES log message is as following:-
elasticsearch.log:[2015-09-16 01:22:37,954][DEBUG][action.bulk ] [ESMasterData2] [jab-prod-reincarnation-v1-2015-09-16][0] failed to execute bulk item (index) index {[jab-prod-reincarnation-v1-2015-09-16][lumberjack][AU_UmY2o7YuwugZ18KCl], source[{"timestamp":"2015-09-16T10:51:54.854+05:30","level":"error","reqId":"39624368-71de-4d47-8103-9f40a4973ba9","sessionId":"cef145fe40d48fefef4430f1270dfaeaf28a12615aca2b01c72c2c508e970a38","content":{"message":"TRACKING DATA POPULATION FAILED","loggedIn":"0","clientRequest":{"url":"/river-island-Black-Free-Item-651511.html","path":"/pdp","method":"GET","originalUrl":"/river-island-Black-Free-Item-651511.html","query":{},"bodyExists":false,"headers":{"connection":"upgrade","host":"www.jab.com","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","user-agent":"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36","accept-language":"en-US,en;q=0.8","cookie":"SITEID=new_site_25; __sonar=18274130205066917707; __utma=93690246.789214969.1442344107.1442344107.1442344107.1; __utmz=93690246.1442344107.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _vz=viz_55f72a1a2a061; referrer=direct","x-waf-network":"DmUF7sMn","jade-true-client-ip":"122.162.248.254","x-akamai-config-log-detail":"true","accept-encoding":"gzip","akamai-origin-hop":"2","via":"1.0 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)","x-forwarded-for":"122.162.248.254, 96.17.182.144, 59.144.112.204, 127.0.0.1","cache-control":"max-age=0","x-akamai-prefetched-object":"1"}},"exception":{"realm":"TypeError","stack":"TypeError: Cannot read property 'name' of undefined\n at Object.PDP (/var/www/html/reincarnation/api/pojo/viewObjects/Tracking/Doubleclick/PDP.js:18:42)\n at /var/www/html/reincarnation/api/models/Tracking.js:45:87\n at Function.forEach (/var/www/html/reincarnation/node_modules/lodash/dist/lodash.js:3298:15)\n at Object.Tracking.getProviderData (/var/www/html/reincarnation/api/models/Tracking.js:40:11)\n at /var/www/html/reincarnation/api/controllers/PDPController.js:50:56\n at tryCatcher (/var/www/html/reincarnation/node_modules/bluebird/js/main/util.js:26:23)\n at Promise._settlePromiseFromHandler (/var/www/html/reincarnation/node_modules/bluebird/js/main/promise.js:503:31)\n at Promise._settlePromiseAt (/var/www/html/reincarnation/node_modules/bluebird/js/main/promise.js:577:18)\n at Async._drainQueue (/var/www/html/reincarnation/node_modules/bluebird/js/main/async.js:128:12)\n at Async._drainQueues (/var/www/html/reincarnation/node_modules/bluebird/js/main/async.js:133:10)\n at Immediate.Async.drainQueues (/var/www/html/reincarnation/node_modules/bluebird/js/main/async.js:15:14)\n at Immediate.wrapped [as _onImmediate] (/var/www/html/reincarnation/node_modules/newrelic/lib/transaction/tracer/index.js:157:28)\n at processImmediate [as _immediateCallback] (timers.js:358:17)","errorCode":{"key":"JBxDefault","value":"Opps! Something went wrong."},"message":"Cannot read property 'name' of undefined"}},"@version":"1","@timestamp":"2015-09-16T05:21:56.108Z","type":"lumberjack","file":"/var/log/reincarnation/reincarnation.log.2015-09-16","host":"reincarnation10","offset":"11700676","app":"reincarnation","env":"prod","iver":"v1"}]}