Split a field in elastic search


(Tim Rice) #1

I have winbeats installed and things are working great. I have a field that is called:
event_data.RuleName

and an actual value is:
technique_id=T1130,technique_name=Install Root Certificate

I would like to split that field into two fields.
One
technique_id and the other technique_name. Each field would have its perspective values.

Would I use something like split field?
{
"split": {
"field": "event_data.RuleName",
"separator": ","
}
}

If so, how do I apply this to any index starting with winlogbeat-*

Thanks,


(Ben Trent) #2

Hey @Tim_Rice

You can definitely do this, I am thinking you would want the kv-processor

Example:

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "processors": [
      {"kv": {
          "field": "event_data.RuleName",
          "field_split": ",",
          "value_split": "="
     }}]
  },
  "docs" : [
    { "_source": {"event_data.RuleName":"technique_id=T1130,technique_name=Install Root Certificate"} }
  ]
}

This will result in:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "technique_name" : "Install Root Certificate",
          "ruleName" : "technique_id=T1130,technique_name=Install Root Certificate",
          "technique_id" : "T1130"
        },
        "_ingest" : {
          "timestamp" : "2019-03-19T19:04:53.588Z"
        }
      }
    }
  ]
}

For creating an ingest pipeline see the put-pipeline-api.

As for applying an IngestPipeline to an index by default, you can set the index.default_pipeline setting.

To apply a default setting to indices matching a pattern use indices-templates

I would look to makes sure that there are no existing templates or pipelines for those types of indices already. You may not want to remove good and existing behavior.


(Tim Rice) #4

Thanks for the assistance. I tried your suggestion, and nothing worked.

I also tried:
PUT _ingest/pipeline/mitresplit
{
"description": "splits technique_name and technique_id",
"processors": [
{
"kv": {
"field": "event_data.RuleName",
"field_split": ",",
"value_split": "="
}
}
]
}


(Ben Trent) #5

@Tim_Rice, I am unsure what you mean by "nothing worked".

Is the pipeline (the one you created mitresplit) associated with the index through the index.default_pipeline option? Also, this will only work on new documents being indexed.


(Tim Rice) #6

I really don't know how to add the pipeline to the index.default_pipeline. I am still trying to research online to see if I can figure this out.

In respects to the "nothing worked", I should have been more descriptive there. Sorry about that.
It appears that if the "field" has a dot in it, I receive a java error. In this example, the field is event_data.RuleName

If I change this to event_dataRulename in your simulation, it works just fine.

The error:

        "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [event_data] not present as part of path [event_data.RuleName]",

(Ben Trent) #7

Heya @Tim_Rice, my simulated pipeline is indeed wrong :). Looking at how the data is structured, it seems that it is of the form

{"event_data":{"ruleName":"technique_id=T1130,technique_name=Install Root Certificate"}}

So, I wrote a new simulation and it works just fine with data in that format:

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "processors": [
      {
        "kv": {
          "field": "event_data.ruleName",
          "field_split": ",",
          "value_split": "="
        }
      }
    ]
  },
  "docs" : [
    { "_source": {"event_data":{"ruleName":"technique_id=T1130,technique_name=Install Root Certificate"}} }
  ]
}

As for updating the ephemeral settings:

PUT winlogbeat-*/_settings/
{
  "index.default_pipeline":"my-pipeline"
}

CAUTION: I am not sure that winlogbeat does not already have a default_pipeline defined, you should be able to see if there is just by looking at the settings.

To make sure that the pipeline is continually set on new winlogbeat-* indices, you can update the existing template (to see templates call GET _cat/templates).


(Tim Rice) #8

BenTrent, YOU ARE THE MAN!!!! Everything works perfectly!


(system) closed #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.