Add_field and copy data

[mathurin68]

Alright, I think I just about have what I'm looking for ...

Any activity that comes over event_data.CommandLine the doesn't have " " quotations around it. I want to rename the field and copy the data from the field into the new one.

This almost seems to work, it creates the new field but it doesn't copy the info over...

filter {
if "(.*?)" not in [event_data.CommandLine] {
mutate {
add_field => {"event_data.Suspicious" => "Suspicious Activity"}
copy => {"event_data.CommandLine" => "event_data.Suspicious" }
}
}
}

This does create the field, event_data.Suspicious but it doesn't copy the data over.

As always I appreciate the help!!

[Badger]

copy => {"[event_data][CommandLine]" => "event_data.Suspicious" }

is what you want. Assuming you want a dot in the field name.

[mathurin68]

Thanks Badger! Why do I have to go 'down the tree' on one and not the other?

Why not this -
copy => {"[event_data][CommandLine]" => "[event_data].[Suspicious]" }

Appreciate it, this looks promising!

[mathurin68]

Also, and this is just a pet peeve thing but the add field puts a , at the end of it.

add_field => {"event_data.Suspicious" => ""}

...a little thing but I was curious.

[Badger]

If you input

{ "Foo": 1, "Bar":"2" }

into logstash with this config

input { stdin { } }
output { stdout { codec => rubydebug } }

filter {
  json {
    source => "message"
    target => "event_data"
  }
  mutate {
  add_field => {"event_data.Suspicious" => "Suspicious Activity"}
  }
  mutate {
  copy => {"[event_data][Foo]" => "event_data.withdot" }
  }
  mutate {
  copy => {"[event_data][Foo]" => "[event_data][withbracket]" }
  }
  mutate {
  }
}

You will see the difference.

{
               "@timestamp" => 2017-11-30T17:25:46.727Z,
                 "@version" => "1",
                     "host" => "[...]",
               "event_data" => {
        "withbracket" => 1,
                "Bar" => "2",
                "Foo" => 1
    },
                  "message" => "{ \"Foo\": 1, \"Bar\":\"2\" }",
    "event_data.Suspicious" => "Suspicious Activity",
       "event_data.withdot" => 1
}

[mathurin68]

Alright! I think I'm staring to get this, if I can trouble you for one last suggestion, as a final step I need to convert the field to keyword(end goal being to use this in machine learning)

From other discussions on here I put this together but can't get it to work and tried all kinds of combos -
filter {
if [event_id] == 4688 and [event_data][CommandLine] !~ /"(.*?)"/ {
mutate {
add_field => {"event_data.Suspicious" => ""}
}
mutate {
copy => {"[event_data][CommandLine]" => "event_data.Suspicious" }
}
mutate {
convert => ["[event_data][Suspicious]", "keyword"]
}
}
}

... keep getting the dreaded pipeline terminated error.

I appreciate it, especially the explanations!

[Badger]

Whether something is a keyword is determined by the template field mapping in Elasticsearch, it is not a logstash thing. If you are using the default logstash-* template pretty much every string field has a .keyword field added to it.

You should ask a new question over in the Elasticsearch forum.

[mathurin68]

Thanks dude, I found it the dev tools, it was already a keyword! Thanks so much for your help!