Help changing Bro logs Epoch time to readable

Elastic Stack Logstash
1

Hey all, thanks for the add.

I've developed my config file that uses filebeat to push bro logs to logstash. Logstash parses pretty well, but my only problem that im running into is that I cannot change the epoch time a readable format. I tried several methods and it never runs correctly; so i changed by the config to original that works. Any suggestions on how to do this?

Here's my code:

input {
beats {
port => 5044
ssl => false
}
}

filter {

#GROK CONN.LOG
if [type] == "CONN_LOG" {
grok {
match => {"message" => "%{NUMBER:ts} %{NOTSPACE:uid} %{IP:orig_h} %{INT:orig_p} %{IP:resp_h}
%{INT:resp_p} %{WORD:proto} %{GREEDYDATA:service} %{NUMBER:duration} %{NUMBER:orig_bytes}
%{NUMBER:resp_bytes} %{GREEDYDATA:conn_state} %{GREEDYDATA:local_orig} %{GREEDYDATA:missed_bytes}
%{GREEDYDATA:history} %{GREEDYDATA:orig_pkts} %{GREEDYDATA:orig_ip_bytes} %{GREEDYDATA:resp_pkts}
%{GREEDYDATA:resp_ip_bytes} %{GREEDYDATA:tunnel_parents}" }
}
}

#GROK HTTP.LOG

else if [type] == "HTTP_LOG" {

grok {

match => {"message" => "%{NUMBER:ts} %{NOTSPACE:uid} %{IP:orig_h} %{INT:orig_p} %{IP:resp_h}

#%{INT:resp_h} %{INT:trans_depth} %{GREEDYDATA:method} %{GREEDYDATA:domain} %{GREEDYDATA:uri}
#%{GREEDYDATA:referrer} %{GREEDYDATA:user_agent} %{NUMBER:request_body_len}
#%{NUMBER:response_body_len} %{GREEDYDATA:status_code} %{GREEDYDATA:status_msg}
#%{GREEDYDATA:info_code} %{GREEDYDATA:info_msg} %{GREEDYDATA:filename} %{GREEDYDATA:bro_tags}
#%{GREEDYDATA:username} %{GREEDYDATA:password} %{GREEDYDATA:proxied} %{GREEDYDATA:orig_fuids}
#%{GREEDYDATA:orig_mime_types} %{GREEDYDATA:resp_fuids} %{GREEDYDATA:resp_mime_types}" }

}

}

#GROK DNS.LOG
else if [type] == "DNS_LOG" {
grok {
match => {"message" => "%{NUMBER:ts} %{NOTSPACE:uid} %{IP:orig_h} %{INT:orig_p} %{IP:resp_h}
%{INT:resp_p} %{WORD:proto} %{INT:trans_id} %{GREEDYDATA:query} %{GREEDYDATA:qclass}
%{GREEDYDATA:qclass_name} %{GREEDYDATA:qtype} %{GREEDYDATA:qtype_name} %{GREEDYDATA:rcode}
%{GREEDYDATA:rcode_name} %{GREEDYDATA:AA} %{GREEDYDATA:TC} %{GREEDYDATA:RD} %{GREEDYDATA:RA}
%{GREEDYDATA:Z} %{GREEDYDATA:answers} %{GREEDYDATA:TTLs} %{GREEDYDATA:rejected}" }
}
}
}

#GROK FILES.LOG

else [type] == "FILE_LOG" {

grok {

match => {"message" => "%{NUMBER:ts} %{NOTSPACE:fuid} %{IP:tx_hosts} %{IP:rx_hosts}

#%{NOTSPACE:conn_uids} %{GREEDYDATA:source} %{GREEDYDATA:depth} %{GREEDYDATA:analyzers}
#%{GREEDYDATA:mime_type} %{GREEDYDATA:filename} %{GREEDYDATA:duration} %{GREEDYDATA:local_orig}
#%{GREEDYDATA:is_orig} %{GREEDYDATA:seen_bytes} %{GREEDYDATA:total_bytes}
#%{GREEDYDATA:missing_bytes} %{GREEDYDATA:overflow_bytes} %{GREEDYDATA:timedout}
#%{GREEDYDATA:parent_fuid} %{GREEDYDATA:md5} %{GREEDYDATA:sha1} %{GREEDYDATA:sha256}
#%{GREEDYDATA:extracted}" }

}

}

#}

#OUTPUT SECTION
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout {
codec => rubydebug
}
}

2

You need a date filter in there to convert it.

Also please use code tags to format things so they are easier to read :slight_smile:

3

Okay, I've updated my code - please observe the CONN log and HTTP log where I added the date filter. When I run through ELK, Kibana still pulls up the time in epoch in the Discover tab. The code doesn't error out though which leads me to believe that is working, i just dont know why the time is not being displayed in a UNIX format.

input {
beats {
port => 5044
ssl => false
}
}

filter {

GROK CONN.LOG - used to parse CONN.LOGs from Bro

if [type] == "CONN_LOG" {
grok {
match => {"message" => "%{NUMBER:ts} %{NOTSPACE:uid} %{IP:orig_h} %{INT:orig_p} %{IP:resp_h}
%{INT:resp_p} %{WORD:proto} %{GREEDYDATA:service} %{NUMBER:duration} %{NUMBER:orig_bytes}
%{NUMBER:resp_bytes} %{GREEDYDATA:conn_state} %{GREEDYDATA:local_orig} %{GREEDYDATA:missed_bytes}
%{GREEDYDATA:history} %{GREEDYDATA:orig_pkts} %{GREEDYDATA:orig_ip_bytes} %{GREEDYDATA:resp_pkts}
%{GREEDYDATA:resp_ip_bytes} %{GREEDYDATA:tunnel_parents}" }
}

added date filter - not sure if it works though

when I pull up kibana the TS field still shows EPOC time

date {
match => ["ts", "UNIX"]
}
}
################################################################

GROK HTTP.LOG - used to parse HTTP.LOGs from BRO

else if [type] == "HTTP_LOG" {
grok {
match => {"message" => "%{NUMBER:ts} %{NOTSPACE:uid} %{IP:orig_h} %{INT:orig_p} %{IP:resp_h}
%{INT:resp_h} %{INT:trans_depth} %{GREEDYDATA:method} %{GREEDYDATA:domain} %{GREEDYDATA:uri}
%{GREEDYDATA:referrer} %{GREEDYDATA:user_agent} %{NUMBER:request_body_len}
%{NUMBER:response_body_len} %{GREEDYDATA:status_code} %{GREEDYDATA:status_msg}
%{GREEDYDATA:info_code} %{GREEDYDATA:info_msg} %{GREEDYDATA:filename} %{GREEDYDATA:bro_tags}
%{GREEDYDATA:username} %{GREEDYDATA:password} %{GREEDYDATA:proxied} %{GREEDYDATA:orig_fuids}
%{GREEDYDATA:orig_mime_types} %{GREEDYDATA:resp_fuids} %{GREEDYDATA:resp_mime_types}" }
}
date {
match => ["ts", "UNIX"]
}
}
################################################################

GROK DNS.LOG - used to parse DNS.LOGs from BRO

else if [type] == "DNS_LOG" {
grok {
match => {"message" => "%{NUMBER:ts} %{NOTSPACE:uid} %{IP:orig_h} %{INT:orig_p} %{IP:resp_h}
%{INT:resp_p} %{WORD:proto} %{INT:trans_id} %{GREEDYDATA:query} %{GREEDYDATA:qclass}
%{GREEDYDATA:qclass_name} %{GREEDYDATA:qtype} %{GREEDYDATA:qtype_name} %{GREEDYDATA:rcode}
%{GREEDYDATA:rcode_name} %{GREEDYDATA:AA} %{GREEDYDATA:TC} %{GREEDYDATA:RD} %{GREEDYDATA:RA}
%{GREEDYDATA:Z} %{GREEDYDATA:answers} %{GREEDYDATA:TTLs} %{GREEDYDATA:rejected}" }
}
date {
match => ["ts", "UNIX"]
}
}
}
###############################################################

The following lines have not been implemented yet, still TSing

#GROK FILES.LOG

else [type] == "FILE_LOG" {

grok {

match => {"message" => "%{NUMBER:ts} %{NOTSPACE:fuid} %{IP:tx_hosts} %{IP:rx_hosts}

#%{NOTSPACE:conn_uids} %{GREEDYDATA:source} %{GREEDYDATA:depth} %{GREEDYDATA:analyzers}
#%{GREEDYDATA:mime_type} %{GREEDYDATA:filename} %{GREEDYDATA:duration} %{GREEDYDATA:local_orig}
#%{GREEDYDATA:is_orig} %{GREEDYDATA:seen_bytes} %{GREEDYDATA:total_bytes}
#%{GREEDYDATA:missing_bytes} %{GREEDYDATA:overflow_bytes} %{GREEDYDATA:timedout}
#%{GREEDYDATA:parent_fuid} %{GREEDYDATA:md5} %{GREEDYDATA:sha1} %{GREEDYDATA:sha256}
#%{GREEDYDATA:extracted}" }

}

}

#}
###############################################################

#OUTPUT SECTION
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout {
codec => rubydebug
}
}

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.