bulkResponse had errors with response statuses:counts of... {

[larryzhu]

We consistently hit the invalid request error in elastic SIEM
Found 10000 signals from the indexes of "[oci-audit-span*]" using signal rule name: "OCI Audit: Delete VCN [Duplicate]", id: "0e2d1191-7600-4268-be75-0f13ce1e2356", rule_id: "4c59e3cc-92d7-4717-a019-da87a1ea66b0", pushing signals to index ".siem-signals-default"

{"type":"log","@timestamp":"2020-04-04T02:32:06Z","tags":["error","plugins","siem"],"pid":8,"message":" bulkResponse had errors with response statuses:counts of...
{
"400": 100
}"}

how to troubleshoot this?

[Frank_Hassanabad]

That's an interesting rule. I don't think that is one of our prepackaged ones but is from another location?

That 400 code happening 100 times looks like it could be an invalid query. I would review the rule in the details section to copy what the query is and then see if the query is valid or not by running the query directly within timeline's KQL search.

I would also see if other rules run correctly against your index of oci-audit-span and this is the only one not running correctly.

[larryzhu]

I created a pull request along with the screenshot:

print detailed bulk response error message

any insights on the version_conflict_engine_exception error here? This happens rather frequently.

[Frank_Hassanabad]

Hi @larryzhu, thanks for the community pull request and the errors on there.

I took the errors and formatted them like so:

[
   {
      "create":{
         "_index":".siem-signals-default-000001",
         "_type":"_doc",
         "_id":"dc2e284becb41e00a76541440aa55d93f2c94a0caf0003a403c80bd5197abaa7",
         "status":409,
         "error":{
            "type":"version_conflict_engine_exception",
            "reason":"[dc2e284becb41e00a76541440aa55d93f2c94a0caf0003a403c80bd5197abaa7]: version conflict, document already exists (current version [1])",
            "index_uuid":"1ZHR_A3PTXevZv49ok4oaQ",
            "shard":"0",
            "index":".siem-signals-default-000001"
         }
      }
   },
   {
      "create":{
         "_index":".siem-signals-default-000001",
         "_type":"_doc",
         "_id":"19bc7fe89cdfa6303812dcce37103cbf5037faeba0fd173c13f428b863a9999f",
         "status":409,
         "error":{
            "type":"version_conflict_engine_exception",
            "reason":"[19bc7fe89cdfa6303812dcce37103cbf5037faeba0fd173c13f428b863a9999f]: version conflict, document already exists (current version [1])",
            "index_uuid":"1ZHR_A3PTXevZv49ok4oaQ",
            "shard":"0",
            "index":".siem-signals-default-000001"
         }
      }
   },
   {
      "create":{
         "_index":".siem-signals-default-000001",
         "_type":"_doc",
         "_id":"30d34bebebd65b06373f65a36961de4c22842c5f968953aa... with response statuses:counts of... { "         400         "\": 2 }"

For your pull request I can leave some comments for you. Could you sign the CLA here?

https://www.elastic.co/contributor-agreement

And make sure that the email you are using for your community pull request is the one that you sign the CLA with?

[Frank_Hassanabad]

@larryzhu,

I left some comments but looking through this, we are truncating right before the important errors that we are interested in which is the 400. The 409 conflicts are not of an interest to us since we perform de-duplication of signals and expect every so often to get 409 conflicts.

To get to the important part which is the 400 errors that we want to see what you want to do is replace your code from your PR with these lines:

    itemsWithErrors
      .filter(item => item.create.status !== 409) // remove any 409 conflict error codes as those are expected due to de-duplication
      .forEach(item => {
        if (item.create.error != null) {
          logger.error(
            `Rule id: "${id}" rule_id: "${ruleParams.ruleId}" name: "${name}" has an error response with the reason of: ${item.create.error.reason}`
          );
        } else {
          // return the full item if there is no reason but typically Elastic Search should always return a reason
          logger.error(
            `Rule id: "${id}" rule_id: "${
              ruleParams.ruleId
            }" name: "${name}" has an error response with the reason of: ${JSON.stringify(item)}`
          );
        }
      });

if (!isEmpty(errorCountsByStatus)) {

It will look like this:

Then you should see in your log files a log line for each Elastic Search reason of why that 400 error message is being hit and that is the important part we want to see in order to understand why your custom rule is not executing well on the server.

[Frank_Hassanabad]

@larryzhu, out of curiosity have you found what your 400 and/or 100 errors were? Did you get the rules you wanted to use to run correctly?