Decoding base64 logs when input is in JSON format?

[Mudabbir]

Greetings !

I am unable to decode my base64 logs that are coming from nxlog server.
Here is my logstash configuration.

The error i am getting is shown below

The log field is in base64
I have tried cipher filters too and also tried it with ruby.
Suggest me possble solutions please

[ylasri (Yassine LASRI)]

Can you avoid sharing screen shots and share your :

• Config file
• Logs

[Mudabbir]

config file:

input {

tcp {

    port => 5142
    type => "ossim-events"
    codec => json {
            charset => "CP1252"
    }

}
}

filter {

mutate {
add_field => {

  "Agent_IP" => "%{host}"
 }

}

######## ALIENVAULT OSSIM Logs ########################################

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} [%{DATA:class}]:%{GREEDYDATA:message}" }
}

ruby {

code => 'event.set("decoded", Base64.decode64(event.get("[message][embedded][field]["log"]")))'
}

if [type] == "ossim-events" {
kv {
value_split => "='"
field_split => "' "
}
}

}

output {

stdout { }

elasticsearch {

    hosts => ["http://localhost:9200"]
    template => ["/var/test/elasticsearch-template.json"]
    template_overwrite => true
    codec => rubydebug

}
}

Logs in console

[2020-11-04T11:24:21,293][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,499 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="7033" plugin_sid="1005402" src_ip="0.0.0.0" dst_ip="0.0.0.0" username="YXZhcGk=" userdata1="YXZhcGk=" userdata2="L2hvbWUvYXZhcGk=" userdata3="L2Jpbi9zaCAtYyAvdXNyL2Jpbi9weXRob24gL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My9hdl9wbHVnaW5zOyBybSAtcmYgL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My8g"[truncated 780 chars]; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,499 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="7033" plugin_sid="1005402" src_ip="0.0.0.0" dst_ip="0.0.0.0" username="YXZhcGk=" userdata1="YXZhcGk=" userdata2="L2hvbWUvYXZhcGk=" userdata3="L2Jpbi9zaCAtYyAvdXNyL2Jpbi9weXRob24gL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My9hdl9wbHVnaW5zOyBybSAtcmYgL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My8gPi9kZXYvbnVsbCAyPiYx" log="QVYgLSBBbGVydCAtICIxNjA0NDgyNTc4IiAtLT4gUklEOiAiNTQwMiI7IFJMOiAiMyI7IFJHOiAic3lzbG9nLHN1ZG8iOyBSQzogIlN1Y2Nlc3NmdWwgc3VkbyB0byBST09UIGV4ZWN1dGVkIjsgVVNFUjogImF2YXBpIjsgU1JDSVA6ICJOb25lIjsgSE9TVE5BTUU6ICJhbGllbnZhdWx0IjsgTE9DQVRJT046ICIvdmFyL2xvZy9hdXRoLmxvZyI7IEVWRU5UOiAiW0lOSVRdTm92ICA0IDE0OjM2OjE2IGFsaWVudmF1bHQgc3VkbzogICAgYXZhcGkgOiBUVFk9cHRzLzcgOyBQV0Q9L2hvbWUvYXZhcGkgOyBVU0VSPXJvb3QgOyBDT01NQU5EPS9iaW4vc2ggLWMgL3Vzci9iaW4vcHl0aG9uIC9ob21lL2F2YXBpLy5hbnNpYmxlL3RtcC9hbnNpYmxlLTE2MDQ0ODI1NzYuMjUtMTUzOTE3NDkwMDk1ODMvYXZfcGx1Z2luczsgcm0gLXJmIC9ob21lL2F2YXBpLy5hbnNpYmxlL3RtcC9hbnNpYmxlLTE2MDQ0ODI1NzYuMjUtMTUzOTE3NDkwMDk1ODMvID4vZGV2L251bGwgMj4mMVtFTkRdIjsg" fdate="2020-11-04 09:36:18" tzone="5.0" event_id="1e8111eb-b1e8-000c-293a-51ec30cbabd5""}
[2020-11-04T11:24:21,294][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,504 Detector [WARNING]: sudo[4005] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,504 Detector [WARNING]: sudo[4005] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,295][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,503 Output [INFO]: idm-event username="YXZhcGk=" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18" rule="IDM_09""; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,503 Output [INFO]: idm-event username="YXZhcGk=" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18" rule="IDM_09""}
[2020-11-04T11:24:21,300][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,507 Detector [WARNING]: ossec-single-line[7007] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,507 Detector [WARNING]: ossec-single-line[7007] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,301][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,506 Detector [WARNING]: sudo[4005] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,506 Detector [WARNING]: sudo[4005] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,301][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,508 Detector [WARNING]: ossec-single-line[7007] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,508 Detector [WARNING]: ossec-single-line[7007] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,302][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,508 Detector [WARNING]: sudo[4005] Event's device field alienvault is not a valid IP.v4/IP.v6 address, falling back to default local."; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,508 Detector [WARNING]: sudo[4005] Event's device field alienvault is not a valid IP.v4/IP.v6 address, falling back to default local."}
[2020-11-04T11:24:21,302][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,511 Output [INFO]: idm-event username="YXZhcGl8JGRvbWFpbg==" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18""; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,511 Output [INFO]: idm-event username="YXZhcGl8JGRvbWFpbg==" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18""}
[2020-11-04T11:24:21,305][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,510 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="4005" plugin_sid="6" src_ip="0.0.0.0" dst_ip="0.0.0.0" log="Tm92ICA0IDE0OjM2OjE4IGFsaWVudmF1bHQgc3VkbzogcGFtX3VuaXgoc3VkbzpzZXNzaW9uKTogc2Vzc2lvbiBjbG9zZWQgZm9yIHVzZXIgcm9vdCA=" fdate="2020-11-04 09:36:18" tzone="5.0" event_id="1e8111eb-b1e8-000c-293a-51ec3005cbab""; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,510 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="4005" plugin_sid="6" src_ip="0.0.0.0" dst_ip="0.0.0.0" log="Tm92ICA0IDE0OjM2OjE4IGFsaWVudmF1bHQgc3VkbzogcGFtX3VuaXgoc3VkbzpzZXNzaW9uKTogc2Vzc2lvbiBjbG9zZWQgZm9yIHVzZXIgcm9vdCA=" fdate="2020-11-04 09:36:18" tzone="5.0" event_id="1e8111eb-b1e8-000c-293a-51ec3005cbab""}

Image from kibana

[ylasri (Yassine LASRI)]

You are using the json codec to read events but your events seems not to be a valid json format. Try a simple config to get the format of your events

input {

tcp {
    port => 5142
    type => "ossim-events"
}
}

filter {}

output {
	stdout { codec => rubydebug	}
}