Split a field in elastic search

[Tim_Rice]

I have winbeats installed and things are working great. I have a field that is called:
event_data.RuleName

and an actual value is:
technique_id=T1130,technique_name=Install Root Certificate

I would like to split that field into two fields.
One
technique_id and the other technique_name. Each field would have its perspective values.

Would I use something like split field?
{
"split": {
"field": "event_data.RuleName",
"separator": ","
}
}

If so, how do I apply this to any index starting with winlogbeat-*

Thanks,

[BenTrent]

Hey @Tim_Rice

You can definitely do this, I am thinking you would want the kv-processor

Example:

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "processors": [
      {"kv": {
          "field": "event_data.RuleName",
          "field_split": ",",
          "value_split": "="
     }}]
  },
  "docs" : [
    { "_source": {"event_data.RuleName":"technique_id=T1130,technique_name=Install Root Certificate"} }
  ]
}

This will result in:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "technique_name" : "Install Root Certificate",
          "ruleName" : "technique_id=T1130,technique_name=Install Root Certificate",
          "technique_id" : "T1130"
        },
        "_ingest" : {
          "timestamp" : "2019-03-19T19:04:53.588Z"
        }
      }
    }
  ]
}

For creating an ingest pipeline see the put-pipeline-api.

As for applying an IngestPipeline to an index by default, you can set the index.default_pipeline setting.

To apply a default setting to indices matching a pattern use indices-templates

I would look to makes sure that there are no existing templates or pipelines for those types of indices already. You may not want to remove good and existing behavior.

[Tim_Rice]

Thanks for the assistance. I tried your suggestion, and nothing worked.

I also tried:
PUT _ingest/pipeline/mitresplit
{
"description": "splits technique_name and technique_id",
"processors": [
{
"kv": {
"field": "event_data.RuleName",
"field_split": ",",
"value_split": "="
}
}
]
}

[BenTrent]

@Tim_Rice, I am unsure what you mean by "nothing worked".

Is the pipeline (the one you created mitresplit) associated with the index through the index.default_pipeline option? Also, this will only work on new documents being indexed.

[Tim_Rice]

I really don't know how to add the pipeline to the index.default_pipeline. I am still trying to research online to see if I can figure this out.

In respects to the "nothing worked", I should have been more descriptive there. Sorry about that.
It appears that if the "field" has a dot in it, I receive a java error. In this example, the field is event_data.RuleName

If I change this to event_dataRulename in your simulation, it works just fine.

The error:

        "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [event_data] not present as part of path [event_data.RuleName]",

[BenTrent]

Heya @Tim_Rice, my simulated pipeline is indeed wrong :). Looking at how the data is structured, it seems that it is of the form

{"event_data":{"ruleName":"technique_id=T1130,technique_name=Install Root Certificate"}}

So, I wrote a new simulation and it works just fine with data in that format:

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "processors": [
      {
        "kv": {
          "field": "event_data.ruleName",
          "field_split": ",",
          "value_split": "="
        }
      }
    ]
  },
  "docs" : [
    { "_source": {"event_data":{"ruleName":"technique_id=T1130,technique_name=Install Root Certificate"}} }
  ]
}

As for updating the ephemeral settings:

PUT winlogbeat-*/_settings/
{
  "index.default_pipeline":"my-pipeline"
}

CAUTION: I am not sure that winlogbeat does not already have a default_pipeline defined, you should be able to see if there is just by looking at the settings.

To make sure that the pipeline is continually set on new winlogbeat-* indices, you can update the existing template (to see templates call GET _cat/templates).

[Tim_Rice]

BenTrent, YOU ARE THE MAN!!!! Everything works perfectly!