Greetings !
I am unable to decode my base64 logs that are coming from nxlog server.
Here is my logstash configuration.
The error i am getting is shown below
The log field is in base64
I have tried cipher filters too and also tried it with ruby.
Suggest me possble solutions please
Can you avoid sharing screen shots and share your :
config file:
input {
tcp {
port => 5142
type => "ossim-events"
codec => json {
charset => "CP1252"
}
}
}
filter {
mutate {
add_field => {
"Agent_IP" => "%{host}"
}
}
######## ALIENVAULT OSSIM Logs ########################################
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} [%{DATA:class}]:%{GREEDYDATA:message}" }
}
ruby {
code => 'event.set("decoded", Base64.decode64(event.get("[message][embedded][field]["log"]")))'
}
if [type] == "ossim-events" {
kv {
value_split => "='"
field_split => "' "
}
}
}
output {
stdout { }
elasticsearch {
hosts => ["http://localhost:9200"]
template => ["/var/test/elasticsearch-template.json"]
template_overwrite => true
codec => rubydebug
}
}
Logs in console
[2020-11-04T11:24:21,293][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,499 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="7033" plugin_sid="1005402" src_ip="0.0.0.0" dst_ip="0.0.0.0" username="YXZhcGk=" userdata1="YXZhcGk=" userdata2="L2hvbWUvYXZhcGk=" userdata3="L2Jpbi9zaCAtYyAvdXNyL2Jpbi9weXRob24gL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My9hdl9wbHVnaW5zOyBybSAtcmYgL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My8g"[truncated 780 chars]; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,499 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="7033" plugin_sid="1005402" src_ip="0.0.0.0" dst_ip="0.0.0.0" username="YXZhcGk=" userdata1="YXZhcGk=" userdata2="L2hvbWUvYXZhcGk=" userdata3="L2Jpbi9zaCAtYyAvdXNyL2Jpbi9weXRob24gL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My9hdl9wbHVnaW5zOyBybSAtcmYgL2hvbWUvYXZhcGkvLmFuc2libGUvdG1wL2Fuc2libGUtMTYwNDQ4MjU3Ni4yNS0xNTM5MTc0OTAwOTU4My8gPi9kZXYvbnVsbCAyPiYx" log="QVYgLSBBbGVydCAtICIxNjA0NDgyNTc4IiAtLT4gUklEOiAiNTQwMiI7IFJMOiAiMyI7IFJHOiAic3lzbG9nLHN1ZG8iOyBSQzogIlN1Y2Nlc3NmdWwgc3VkbyB0byBST09UIGV4ZWN1dGVkIjsgVVNFUjogImF2YXBpIjsgU1JDSVA6ICJOb25lIjsgSE9TVE5BTUU6ICJhbGllbnZhdWx0IjsgTE9DQVRJT046ICIvdmFyL2xvZy9hdXRoLmxvZyI7IEVWRU5UOiAiW0lOSVRdTm92ICA0IDE0OjM2OjE2IGFsaWVudmF1bHQgc3VkbzogICAgYXZhcGkgOiBUVFk9cHRzLzcgOyBQV0Q9L2hvbWUvYXZhcGkgOyBVU0VSPXJvb3QgOyBDT01NQU5EPS9iaW4vc2ggLWMgL3Vzci9iaW4vcHl0aG9uIC9ob21lL2F2YXBpLy5hbnNpYmxlL3RtcC9hbnNpYmxlLTE2MDQ0ODI1NzYuMjUtMTUzOTE3NDkwMDk1ODMvYXZfcGx1Z2luczsgcm0gLXJmIC9ob21lL2F2YXBpLy5hbnNpYmxlL3RtcC9hbnNpYmxlLTE2MDQ0ODI1NzYuMjUtMTUzOTE3NDkwMDk1ODMvID4vZGV2L251bGwgMj4mMVtFTkRdIjsg" fdate="2020-11-04 09:36:18" tzone="5.0" event_id="1e8111eb-b1e8-000c-293a-51ec30cbabd5""}
[2020-11-04T11:24:21,294][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,504 Detector [WARNING]: sudo[4005] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,504 Detector [WARNING]: sudo[4005] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,295][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,503 Output [INFO]: idm-event username="YXZhcGk=" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18" rule="IDM_09""; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,503 Output [INFO]: idm-event username="YXZhcGk=" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18" rule="IDM_09""}
[2020-11-04T11:24:21,300][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,507 Detector [WARNING]: ossec-single-line[7007] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,507 Detector [WARNING]: ossec-single-line[7007] Event's field src_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,301][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,506 Detector [WARNING]: sudo[4005] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,506 Detector [WARNING]: sudo[4005] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,301][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,508 Detector [WARNING]: ossec-single-line[7007] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,508 Detector [WARNING]: ossec-single-line[7007] Event's field dst_ip alienvault is not a valid IP.v4/IP.v6 address, falling back to default: 0.0.0.0"}
[2020-11-04T11:24:21,302][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,508 Detector [WARNING]: sudo[4005] Event's device field alienvault is not a valid IP.v4/IP.v6 address, falling back to default local."; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,508 Detector [WARNING]: sudo[4005] Event's device field alienvault is not a valid IP.v4/IP.v6 address, falling back to default local."}
[2020-11-04T11:24:21,302][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,511 Output [INFO]: idm-event username="YXZhcGl8JGRvbWFpbg==" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18""; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,511 Output [INFO]: idm-event username="YXZhcGl8JGRvbWFpbg==" hostname="alienvault.alienvault" ip="192.168.50.90" inventory_source="18""}
[2020-11-04T11:24:21,305][WARN ][logstash.codecs.jsonlines][main][9ca85e29afa4d481f486364c739c5065899b0274ee9bc49283e78eb315e8ca5a] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): Expected space separating root-level values
at [Source: (String)"2020-11-04 14:36:18,510 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="4005" plugin_sid="6" src_ip="0.0.0.0" dst_ip="0.0.0.0" log="Tm92ICA0IDE0OjM2OjE4IGFsaWVudmF1bHQgc3VkbzogcGFtX3VuaXgoc3VkbzpzZXNzaW9uKTogc2Vzc2lvbiBjbG9zZWQgZm9yIHVzZXIgcm9vdCA=" fdate="2020-11-04 09:36:18" tzone="5.0" event_id="1e8111eb-b1e8-000c-293a-51ec3005cbab""; line: 1, column: 6]>, :data=>"2020-11-04 14:36:18,510 Output [INFO]: event type="detector" date="1604482578" device="192.168.50.90" interface="eth1" plugin_id="4005" plugin_sid="6" src_ip="0.0.0.0" dst_ip="0.0.0.0" log="Tm92ICA0IDE0OjM2OjE4IGFsaWVudmF1bHQgc3VkbzogcGFtX3VuaXgoc3VkbzpzZXNzaW9uKTogc2Vzc2lvbiBjbG9zZWQgZm9yIHVzZXIgcm9vdCA=" fdate="2020-11-04 09:36:18" tzone="5.0" event_id="1e8111eb-b1e8-000c-293a-51ec3005cbab""}
Image from kibana
You are using the json codec to read events but your events seems not to be a valid json format. Try a simple config to get the format of your events
input {
tcp {
port => 5142
type => "ossim-events"
}
}
filter {}
output {
stdout { codec => rubydebug }
}This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.