Hi,
i totally new to the elastic stack, and currently working on a dash board for logs through kibana, but i'm still at the data ingestion stage unfortunately !
took me some days to learn how to make things work, frist with only grok then with dissect , my problem is i have a line (the first one) containing the date of the logs and the timestamps are witout a date, so i tried using grok for this first line to extract the date then using the add_field option create an aggregation of date and time to construct my full timestamps.
input {
file {
path => "mypath/log"
start_position => "beginning"
sincedb_path => "mypath/file"
}
}
filter {
if [message] =~ /$^\s*$/{
drop{}
}
if "--" in [message] {
grok{
match => { "message" => "%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year}%GREEDYDATA}"}
add_field => { "time" => "%{day}/%{month}/%{year}-%{time_since_startup}"}
}
}
else {
dissect {
mapping => {
"message" => "%{sequence_id} %{time_since_startup} %{process_id} %{puid} %stack_level} %{operation} %{params} %{op_type} %{form_event} %{log_time}"
}
convert_datatype => {
"sequence_id" => "int"
"process_id" => "int"
"puid" => "int"
"stack_level" => "int"
"op_type" => "int"
}
}
date {
match => ["time", "dd/MMM/yyyy-HH:mm:ss:SSS"]
target => "time"
remove_field => "@timestamp"
remove_field => "time_since_startup"
}
}
}
output {
file {
path => "mypath/DebugOut"
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
index => "debug_log_tabular"
}
}
the output doesn't contain my new field andi can't get it to work. thanks in advance for your help here is an example of the output :
{
"sequence_id" => 7,
"process_id" => 7,
"puid" => 7,
"form_event" => "0",
"log_time" => "\r",
"path" => "mypath",
"op_type" => 1,
"stack_level" => 0,
"params" => "Faux",
"operation" => "34",
"message" => "7\t15:39:43:246\t7\t7\t0\t34\tFaux\t1\t0\t\r",
"time_since_startup" => "15:39:43:246",
"@timestamp" => 2020-10-26T13:02:04.292Z,
"host" => "myMachine",
"@version" => "1"
}
also i would love to know how the timestamp is still there even if i deleted it explicitly .