Elastic SIEM

[Elk_huh (Brian)]

I am having issues starting the ML datafeeds for SIEM, I've tried manually putting in a mapping for @timestamp as date but i still get this error
anyone else have this issue ?
` "{"error":{"root_cause":[{"type":"status_exception","reason":"[datafeed-rare_process_by_host_windows_ecs] cannot retrieve field [@timestamp] because it has no mappings"}],"type":"status_exception","reason":"[datafeed-rare_process_by_host_windows_ecs] cannot retrieve field [@timestamp] because it has no mappings"},"status":400}"

[Andrew_G (Andrew Macri)]

Hi @Elk_huh!

Another person who encountered a similar error found the mapping issue via the instructions here.

Would you be willing to follow the steps suggested in the link above and post the output if those steps don't help resolve the error?

[Elk_huh (Brian)]

@Andrew_G

be willing to follow t

So one thing i noticed is the ML jobs created by the detections tab is looking for winlogbeat-* index, we are using a custom index called something different which is specified in kibana --> advanced settings --> SIEM ,

I changed the index to winlogbeat just to see if it fixed it , no dice
Please see field mappings

GET winlogbeat-7*/_mapping
{
"winlogbeat-7-2020.10.13" : {
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"deviceproduct" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"devicevendor" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"event" : {
"properties" : {
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"code" : {
"type" : "long"
},
"created" : {
"type" : "date"
},
"kind" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"provider" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"host" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"log" : {
"properties" : {
"level" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"winlog" : {
"properties" : {
"api" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"channel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"computer_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"event_data" : {
"properties" : {
"AccessList" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"AccessMask" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"Binary" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"HandleId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ObjectName" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ObjectServer" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ObjectType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"PrivilegeList" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ProcessId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ResourceAttributes" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"SubjectDomainName" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"SubjectLogonId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"SubjectUserName" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"SubjectUserSid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"TransactionId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"param1" : {

[Andrew_G (Andrew Macri)]

Thanks for your reply @Elk_huh!

I attempted to reproduce the issue you reported by creating a new instance of 7.9.2 in Elastic Cloud, and ingesting some data from Winlogbeat. I wasn't able to reproduce it, but I documented the steps I took so we have a working environment to compare with yours.

Running winlogbeat setup

After creating a new deployment, I ran winlogbeat setup on a Windows host:

 C:\Program Files\Winlogbeat> .\winlogbeat.exe setup

and then started ingesting data from the WIndows host via:

C:\Program Files\Winlogbeat> .\winlogbeat.exe -e

@Elk_huh, do you recall running winlogbeat.exe setup, and if not, would you be willing to run it?

Enabling the rare_process_by_host_windows_ecs ML Job

Next, I enabled the rare_process_by_host_windows_ecs via the ML job settings shown in the screenshot below:

Activating the Unusual Process For a Windows Host detection rule

After enabling the rare_process_by_host_windows_ecs ML job via the screenshot above, I activated the Unusual Process For a Windows Host detection rule, shown in the screenshot below:

The Detection rules > Monitoring tab shown in the screenshot below reported the rule successfully ran:

Verifying the datafeeds via GET _ml/datafeeds

To see the ML data feeds that were created for the enabled ML jobs, ran the following from Dev Tools:

GET _ml/datafeeds

All of the datafeeds returned for the above query have winlogbeat-* indices, per the sample output below:

{
  "count" : 10,
  "datafeeds" : [
    {
      "datafeed_id" : "datafeed-rare_process_by_host_windows_ecs",
      "job_id" : "rare_process_by_host_windows_ecs",
      "query_delay" : "86021ms",
      "indices" : [
        "winlogbeat-*"
      ],
     ...

@Elk_huh regarding your comment:

So one thing i noticed is the ML jobs created by the detections tab is looking for winlogbeat-* >index, we are using a custom index called something different which is specified in kibana --> >advanced settings --> SIEM ,

I changed the index to winlogbeat just to see if it fixed it , no dice

@Elk_huh would you be willing to verify all of the datafeeds returned by running GET _ml/datafeeds in your environment now specify winlogbeat-*?

Winlogbeat mappings

Having verified that all the ML jobs specify winlogbeat-*, I ran the following in Dev Tools:

GET winlogbeat-*/_mapping

I was hoping to use an online JSON diff to compare the output of GET winlogbeat-*/_mapping in my environment with the output in your reply above, but it looks like the JSON in your post is incomplete. (I'm guessing you hit the character limit on a post.)

That said, the mapping for @timestamp in my environment looks the same as yours:

      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },

A note about required ECS fields when not using Beats

If you're using a custom index for the rare_process_by_host_windows_ecs ML job, the following documentation specifies the required ECS fields when not using Beats.

[Elk_huh (Brian)]

@Andrew_G

the rare_process_by_host_windows_ecs ML job via the screenshot above, I activated the Unusua

Hi Andrew,
thanks for the detailed instructions, Heres the problem, after some troubleshooting, I had to add every single ECS mapping to it even if it isnt used,
that was 1 problem if i used the winlogbeat-* index, and after i start the Ml job, nothing shows up in the detections tab

If i use a custom index specified in kibana --> advanced --> SIEM, the ML jobs for detections is still only looking at winlogbeat-* and not my custom index. This is the part im not sure how to solve

[Andrew_G (Andrew Macri)]

@Elk_huh

If i use a custom index specified in kibana --> advanced --> SIEM, the ML jobs for detections is still only looking at winlogbeat-* and not my custom index. This is the part im not sure how to solve

Let's walk through an end-to-end example where we:

• Configure Winlogbeat to output to a custom index
• Add the custom index to the indexes used by the Security app
• Update the ML job's datafeed to read the custom index

Configure winlogbeat to output to a custom index

To configure winlogbeat to output to a custom index, I edited winlogbeat.yml to add the entries below:

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "custom-winlogbeat"
setup.ilm.pattern: "{now/d}-000001" 
output.elasticsearch.index: "custom-winlogbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "custom-winlogbeat"
setup.template.pattern: "custom-winlogbeat-*"

Next I ran winlogbeat setup on the command line:

PS C:\program files\Winlogbeat> .\winlogbeat.exe setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

Finally, I ran:

PS C:\program files\Winlogbeat> .\winlogbeat.exe -e

on the command line to collect some data.

Installing Sysmon for Process creation events

Since we're using the rare_process_by_host_windows_ecs in our example, I also downloaded and installed Sysmon on the host running Winlogbeat.

I configured Sysmon to use SwiftOnSecurity's sysmon config to capture process creation events:

.\sysmon64.exe -accepteula -i sysmonconfig-export.xml

I started notepad.exe a few times on the windows host to generate some process creation events.

Configuring the Security app to use the custom index

In Kibana Stack Management, click Kibana Advanced Settings, and select the Security Solution category, as shown in the screenshot below:

The securitySolution:defaultIndex setting specifies the Elasticsearch indices used by the Security app.

In 7.9.2, the default indices are:

apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, packetbeat-*, winlogbeat-*

To ensure we're working with our custom index, let's change

winlogbeat-*

to

custom-winlogbeat-*

so the new securitySolution:defaultIndex setting is:

apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, packetbeat-*, custom-winlogbeat-*

as shown in the screenshot below:

After making the changes above, click the Save Changes button, and reload the page when prompted.

Confirming the Security app is using the custom index

Navigate to the Security app, and click the Timeline button to open Timeline.

Run the following KQL query in Timeline:

_index: custom-winlogbeat-*

and expand an event in the search result. As shown in the screenshot below, we now have confirmation the Security app is using our custom Winlogbeat index:

Verify Process creation events are being ingested

As noted earlier, I installed Sysmon on the Windows host running Winlogbeat to capture process creation events, which are used by the ML job.

To verify process creation events were being ingested, I ran the following query in a timeline:

"event.action": "Process Create (rule: ProcessCreate)" and "agent.type": "winlogbeat"

The query above is similar to the query specified in the rare_process_by_host_windows_ecs's datafeed. The screenshot of Timeline below shows the output of that query:

Note: If you are generating your own events for use with the rare_process_by_host_windows_ecs ML job, the agent.type field must have the value winlogbeat in order to match the query in the rare_process_by_host_windows_ecs job.

Updating the ML job's datafeed to read the custom index

In my previous post, I enabled the rare_process_by_host_windows_ecs ML job and activiated the Unusual Process For a Windows Host detection rule shown in the screenshot below:

Click on the link in the screenshot above to display the rare_process_by_host_windows_ecs ML job, as shown in the screenshot below:

Expand the rare_process_by_host_windows_ecs job shown above, then select the Datafeed tab, shown in the screenshot below:

Next, we're going to modify the indices for the datafeed shown in the screenshot above.

Update the ML job datafeed to use the custom index

In Kibana Dev Tools, run the following query:

GET _ml/datafeeds/datafeed-rare_process_by_host_windows_ecs

The above query will return the datafeed for our ML job:

{
  "count" : 1,
  "datafeeds" : [
    {
      "datafeed_id" : "datafeed-rare_process_by_host_windows_ecs",
      "job_id" : "rare_process_by_host_windows_ecs",
      "query_delay" : "86021ms",
      "indices" : [
        "winlogbeat-*"
      ],
      "query" : {
        "bool" : {
          "filter" : [
            {
              "term" : {
                "event.action" : "Process Create (rule: ProcessCreate)"
              }
            },
            {
              "term" : {
                "agent.type" : "winlogbeat"
              }
            }
          ]
        }
      },
      "scroll_size" : 1000,
      "chunking_config" : {
        "mode" : "auto"
      },
      "delayed_data_check_config" : {
        "enabled" : true
      },
      "max_empty_searches" : 10,
      "indices_options" : {
        "expand_wildcards" : [
          "open"
        ],
        "ignore_unavailable" : false,
        "allow_no_indices" : true,
        "ignore_throttled" : true
      }
    }
  ]
}

Next we're going to update the indices for this datafeed, as documented here to use our custom index.

Execute the following in Dev Tools:

POST _ml/datafeeds/datafeed-rare_process_by_host_windows_ecs/_update
{
  "indices" : [
    "custom-winlogbeat-*"
  ]
}

The response returned by the API will contain the updated indices:

{
  "datafeed_id" : "datafeed-rare_process_by_host_windows_ecs",
  "job_id" : "rare_process_by_host_windows_ecs",
  "query_delay" : "86021ms",
  "indices" : [
    "custom-winlogbeat-*"
  ],
  "query" : {
    "bool" : {
      "filter" : [
        {
          "term" : {
            "event.action" : "Process Create (rule: ProcessCreate)"
          }
        },
        {
          "term" : {
            "agent.type" : "winlogbeat"
          }
        }
      ]
    }
  },
  "scroll_size" : 1000,
  "chunking_config" : {
    "mode" : "auto"
  },
  "delayed_data_check_config" : {
    "enabled" : true
  },
  "max_empty_searches" : 10,
  "indices_options" : {
    "expand_wildcards" : [
      "open"
    ],
    "ignore_unavailable" : false,
    "allow_no_indices" : true,
    "ignore_throttled" : true
  }
}

Re-run the job

Navigate back to the Security App Detections page, and if it's already running, disable and re-enable the rare_process_by_host_windows_ecs ML job, per the screenshot below:

After re-enabling the job, click on rare_process_by_host_windows_ecs to view the Anomaly detection job.

Expand the rare_process_by_host_windows_ecs and click the Datafeed tab. The Datafeed tab will show the updated indices, per the screenshot below:

The screenshot above shows the ML job is using our custom-winlogbeat-* index, and it processed all the events we expected it to.

Tip: You may stop the datafeed via ... menu shown in the screenshot above and re-start it from a specific time to reprocess data, as shown in the screenshot below: