Logstash Translate Dictionary not working

AliMalik · May 19, 2020, 11:03pm

Hi All, I need some help. I am trying to use a translate filter in logstash to translate winlog.event_data.AccessList value to some meaningful field but it's not working. below is my code in the filter file.
Looks like this if statement "if "AccessList" in [event_date]" is not working. I tried [winlog][even_data] and [winlog.event_data] but still doesn't work.

filter {
 if "fileserver" in [tags] {
   if "AccessList" in [event_data] {
    mutate{
           add_field => { "TestField" => "Testing" }
    }
    translate {
      field => "AccessList"
      destination => "Accesses"
      exact => false
      dictionary => [ '%%1537', "Delete",
                      '%%1538', "ReadControl",
                      '%%1539', "ReadControl",
                      '%%1540', "ReadControl",
                      '%%1541', "Synchronize",
                      '%%1542', "Synchronize",
                      '%%4416', "ReadData",
                      '%%4417', "WriteData",
                      '%%4418', "AppendData",
                      '%%4419', "ReadEA",
                      '%%4420', "WriteEA",
                      '%%4423', "ReadAttrib",
                      '%%4424', "WriteAttrib",
                      '%%1801', "Granted",
                      '%%1805', "NotGranted" ]
    }
   }
  }
}

andres-perez · May 20, 2020, 3:32pm

Hi,
Logstash uses bracket notation for fields: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

If you want to check if "winlog.event_data.AccessList" field exists, you must use the condition:

if [winlog][event_data][AccessList] {

AliMalik · May 20, 2020, 9:01pm

Thanks Andres, I have tried this as well but no luck

Badger · May 20, 2020, 9:03pm

If you add

output { stdout { codec => rubydebug } }

what does an event look like? Also what does your modified configuration look like?

AliMalik · May 20, 2020, 9:04pm

give me sec I will try this. I tried using mutate and added a field before this if condition, also i removed if condition but still dictionary didn't work.

AliMalik · May 20, 2020, 9:10pm

Below is the config file but with added output tag logstash service not starting.
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :excep...after filter...

filter {
 if "fileserver" in [tags] and [event_id] == 4663 {
   if [winlog][event_data][AccessList] {
    mutate{
           add_field => { "QaisarAli" => "Testing" }
    }
    translate {
      field => "[winlog][event_data][AccessList]"
      destination => "[ActionTaken]"
      exact => false
      dictionary => [ '%%1537', "Delete",
                      '%%1538', "ReadControl",
                      '%%1539', "ReadControl",
                      '%%1540', "ReadControl",
                      '%%1541', "Synchronize",
                      '%%1542', "Synchronize",
                      '%%4416', "ReadData",
                      '%%4417', "WriteData",
                      '%%4418', "AppendData",
                      '%%4419', "ReadEA",
                      '%%4420', "WriteEA",
                      '%%4423', "ReadAttrib",
                      '%%4424', "WriteAttrib",
                      '%%1801', "Granted",
                      '%%1805', "NotGranted" ]
    }
   }
  }
output { stdout { codec => rubydebug } }
}

AliMalik · May 20, 2020, 9:12pm

sorry i added output on wrong spot, now added outside filter tag when checking the status of logstash it shows log fields. and in discovery i can't see that field that I am adding for Testing. which means that if condition is still not checking.

AliMalik · May 20, 2020, 9:17pm

and in discovery i can't see that field that I am adding for Testing. which means that if condition is still not checking.

AliMalik · May 20, 2020, 9:35pm

# /usr/share/logstash/bin/logstash -f 12-fileaccess-filter.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-05-21 07:33:29.522 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-05-21 07:33:29.533 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.2"}
[INFO ] 2020-05-21 07:33:32.769 [Converge PipelineAction::Create<main>] Reflections - Reflections took 42 ms to scan 1 urls, producing 20 keys and 40 values
[WARN ] 2020-05-21 07:33:34.611 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2020-05-21 07:33:34.616 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/12-fileaccess-filter.conf"], :thread=>"#<Thread:0x77ae9b26 run>"}
[INFO ] 2020-05-21 07:33:35.697 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2020-05-21 07:33:35.762 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-05-21 07:33:36.032 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-21 07:33:36.389 [LogStash::Runner] runner - Logstash shut down.

Badger · May 20, 2020, 10:03pm

It would be really helpful to see an event output by

output { stdout { codec => rubydebug } }

That will allow us to confirm that the field has the name you think it has, that [event_id] is numeric, and not a string, and that "fileserver" is in [tags].

AliMalik · May 20, 2020, 10:17pm

"tags" => [
[0] "eventlog",
[1] "fileserver",
[2] "lmfileserver",
[3] "beats_input_codec_plain_applied"
],
"agent" => {
"ephemeral_id" => "b5f0777f-b027-48bd-989f-08615a66f373",
"type" => "winlogbeat",
"hostname" => "LMFILESERVER",
"version" => "7.6.2",
"id" => "f992d4c6-0696-4d16-81de-7752cd2d31c7"
},
"ecs" => {
"version" => "1.4.0"
},
"winlog" => {
"api" => "wineventlog",
"keywords" => [
[0] "Audit Success"
],
"task" => "File System",
"channel" => "Security",
"event_data" => {
"ObjectServer" => "Security",
"ResourceAttributes" => "S:AI",
"SubjectUserName" => "vtay",
"ObjectName" => "\\Device\\HarddiskVolume7\\Share\\Text\\Business Management System",
"SubjectLogonId" => "0x21296c18",
"HandleId" => "0x1648",
"AccessMask" => "0x20000",
"ProcessId" => "0x4",
"AccessList" => "%%1538\n\t\t\t\t",
"ObjectType" => "File",
"SubjectDomainName" => "WEBVIOUS",
"SubjectUserSid" => "S-1-5-21-1234567897-1234567897-3126549871-1759"
},
"provider_name" => "Microsoft-Windows-Security-Auditing",
"version" => 1,
"process" => {
"pid" => 4,
"thread" => {
"id" => 1804
}
},
"event_id" => 4663,
"computer_name" => "fileserver.webvious.com.au",
"provider_guid" => "{54849625-5478-4994-a5ba-3e3b0328c30d}",
"record_id" => 42186209,
"opcode" => "Info"
},
"@timestamp" => 2020-05-20T03:03:09.967Z,
"event" => {
"created" => "2020-05-20T22:11:23.536Z",
"code" => 4663,
"action" => "File System",
"provider" => "Microsoft-Windows-Security-Auditing",
"kind" => "event"
},
"@version" => "1"
}
{
"host" => {
"architecture" => "x86_64",
"name" => "fileserver.webvious.com.au",
"hostname" => "LMFILESERVER",
"os" => {
"platform" => "windows",
"name" => "Windows Server 2019 Standard",
"version" => "10.0",
"build" => "17763.1158",
"family" => "windows",
"kernel" => "10.0.17763.1158 (WinBuild.160101.0800)"
},

AliMalik · May 20, 2020, 10:28pm

if i only checkt his if "fileserver" in [tags] then it goes to second line if it finds fileserver in tags but when i type this if "fileserver" in [tags] and [winlog][event_id] == "4663" it doesn't do anything so [winlog][event_id] == "4663" doesn't works. i tried removing winlog and removing quotes from 4663 but no luck.

Badger · May 20, 2020, 11:17pm

"event_id" => 4663

There are no quotes around event_id so you should be comparing it to a number, not a string. I would have expected

if "fileserver" in [tags] and [winlog][event_id] == 4663 {
    if [winlog][event_data][AccessList] {

to result in the code in the if block getting executed.

AliMalik · May 20, 2020, 11:21pm

Yes Badger i figured it out 10 minutes ago. you are right by using [winlog][event_id] it worked fine.

Thanks a lot for your help mate.

AliMalik · May 21, 2020, 12:34am

One more thing, Now as I have got this dictionary working and it's translating event id to eventdescription I can see this EventDesc field in discovery section with a question mark infront of it but i can't use this field in Visualizations. how do i do this?

Badger · May 21, 2020, 12:57am

I think that is a kibana question rather than a logstash question, but the answer may be to do a refresh of the fields in the index pattern management page in kibana. I have not run kibana in a long time so I do not recall it very clearly.

AliMalik · May 21, 2020, 1:41am

Badger you are the legend. Got it working now. Thanks a lot

system · June 18, 2020, 1:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.