Ruby Filter - Event.Remove Inconsistent Results

Okay, I can get the debug data now. I had to enable a port via Docker. Found here.

Here is an example of the data I am getting from a sample event. I've redacted some sensitive information.

The raw output is below. However, I believe the interesting piece is here:

[2018-07-05T14:55:33,217][DEBUG][logstash.pipeline        ] output received {"event"=>{"@timestamp"=>2018-07-05T14:55:33.039Z, "@version"=>"1", "event_data"=>{"os_process"=>"51118512", "event_timestamp"=>2018-07-05T14:52:31.045Z, "audit_type"=>"Standard", "object_name"=>"UserTEST", "action_name"=>"DROP USER", "unified_audit_policies"=>"ORA_ACCOUNT_MGMT, ORA_SECURECONFIG", "system_privilege_used"=>"DROP USER", "logType"=>"Oracle", "sql_text"=>"DROP USER UserTEST\u0000", "sessionid"=>#<BigDecimal:60d634bd,'0.272393717E9',9(12)>, "fga_policy_name"=>nil, "return_code"=>#<BigDecimal:5ecfe7b1,'0.0',1(4)>, "userhost"=>"ubuntu", "dbusername"=>"SVC_User_AUDIT1", "external_userid"=>nil, "insertTime"=>2018-07-05T14:55:33.000Z, "os_username"=>"User1234"}}}

You can see that I am able to put everything under the event_data nested object, but no data is being submitted into the ES.

I am still not able to see anything in the dead letter queue. So I don't see any error while this wouldn't be able to submit to ES.

Partial Output (Removed due length requirements):


[2018-07-05T14:55:33,129][DEBUG][logstash.pipeline        ] filter received {"event"=>{"event_timestamp"=>2018-07-05T14:52:30.162Z, "audit_type"=>"Standard", "object_name"=>"UserTEST", "action_name"=>"CREATE USER", "unified_audit_policies"=>"ORA_ACCOUNT_MGMT, ORA_SECURECONFIG", "system_privilege_used"=>"CREATE USER", "@timestamp"=>2018-07-05T14:55:33.023Z, "@version"=>"1", "sql_text"=>"CREATE USER UserTEST IDENTIFIED BY NULL", "sessionid"=>#<BigDecimal:311bce39,'0.272393717E9',9(12)>, "logType"=>"Oracle", "fga_policy_name"=>nil, "return_code"=>#<BigDecimal:462051e1,'0.0',1(4)>, "userhost"=>"ubuntu", "dbusername"=>"SVC_User_AUDIT1", "external_userid"=>nil, "os_username"=>"User1234", "os_process"=>"51118512"}}
[2018-07-05T14:55:33,132][DEBUG][logstash.pipeline        ] output received {"event"=>{"@timestamp"=>2018-07-05T14:55:32.998Z, "@version"=>"1", "event_data"=>{"os_process"=>"51118512", "event_timestamp"=>2018-07-05T14:50:51.705Z, "audit_type"=>"Standard", "object_name"=>"UserTEST", "action_name"=>"DROP USER", "unified_audit_policies"=>"ORA_ACCOUNT_MGMT, ORA_SECURECONFIG", "system_privilege_used"=>"DROP USER", "logType"=>"Oracle", "sql_text"=>"DROP USER UserTEST\u0000", "sessionid"=>#<BigDecimal:6cb8ad51,'0.272393717E9',9(12)>, "fga_policy_name"=>nil, "return_code"=>#<BigDecimal:7346eb29,'0.0',1(4)>, "userhost"=>"ubuntu", "dbusername"=>"SVC_User_AUDIT1", "external_userid"=>nil, "insertTime"=>2018-07-05T14:55:33.000Z, "os_username"=>"User1234"}}}