EPOC time issue

Elastic Stack Logstash
1

Sample Data -- (Data is in CSV file) and provided samples.

25200,"1171007000040000","nzapap176:KUX","1171007000040000",40960,24618,16342,+000000000000000000000000000060.2,+000000000000000000000000000039.8,16384,52,16332,+000000000000000000000000000000.3,+000000000000000000000000000099.7,24576,24502,74,+000000000000000000000000000099.7,+000000000000000000000000000000.3,10986,0,4,29,0,0,15,118,29,6,4,1,112,27,29,27,11,44,138,28,11,8,-1,-1,-1,-000000000000000000000000000000.1,-000000000000000000000000000000.1,1379173,4893346,0,0,8,34,100,0,0,0,+000000000000000000000000000077.7,+000000000000000000000000000021.9,+000000000000000000000000000022.2,1,1,1,1,5387,19114,5461,+000000000000000000000000000063.4,1,+000000000000000000000000000014.4,1,+000000000000000000000000000022.8,1,+000000000000000000000000000005.0,1,+000000000000000000000000000090.0,1,-1,0,-1,0,-1,0,16384,1,52,1,16332,1,+000000000000000000000000000000.3,1,+000000000000000000000000000099.7,1,+000000000000000000000000000001.5,1,4096,1

Logstash output :

"Total_Real_Mem_MB" => "24576",
"ADSS_WHSC" => "1",
"Net_Memory_Avail" => "-1",
"Page_In_15Min" => "0",
"SP_WHSC" => "0",
"path" => "/home/elkuser/KP/LogData/netcool-nzapap176/nzapap176_mem1.csv",
"Comp_Mem_Pct" => "+000000000000000000000000000089.5",
"Page_Out_Reqs" => "0",
"TDSS_WHSC" => "1",
"RMAVA_WHSC" => "1",
"Paging_Space_Write_per_Sec" => "0",
"File_Repl_Mem_Pct" => "+000000000000000000000000000010.5",
"Page_Ins" => "0",
"Page_Out_60Min" => "17",
"SYMEP_WHSC" => "1",
"Decay_Rate" => "0",
"Used_Real_Mem_Pct" => "+000000000000000000000000000099.7",
"Scanned_Pages" => "-1",
"MEMUS_WHSC" => "1",
"Page_Scan" => "0",
"Page_Scan_1Min" => "0",
"UDSSP_WHSC" => "1",
"Repaging_Rate" => "0",
"Page_Scan_15Min" => "0",
"Avail_Disk_Swap_Space_MB" => "16326",
"System_Name" => "nzapap176:KUX",
"System_Paging" => "-1",
"Timestamp" => 39078-04-11T00:48:23.000Z,
"Avail_Virtual_Storage_MB" => "16359",
"Used_Disk_Swap_Space_MB" => "58",
"Avail_Swap_Space_Pct" => "+000000000000000000000000000099.7",
"SYSP_WHSC" => "0",
"Page_Faults" => "2891",
"Filesys_Avail_Mem_Pct" => "+000000000000000000000000000010.4",
"WRITETIME" => 39078-04-11T00:48:23.000Z,
"Page_Scan_60Min" => "0",

Config File:

input {
file {
path => "/home/elkuser/KP/LogData/netcool-nzapap176/nzapap176_mem1.csv"
start_position => "beginning"
sincedb_path => "/dev/null"

codec => multiline {

pattern => "^\A%{NUMBER},%{QUOTEDSTRING},%{QUOTEDSTRING},"

negate => "true"

what => "previous"

max_lines => 1000

}

          }

}
filter{

    csv{

columns => ["TMZDIFF", "WRITETIME", "System_Name", "Timestamp", "Total_Virtual_Storage_MB", "Used_Virtual_Storage_MB", "Avail_Virtual_Storage_MB", "Virtual_Storage_Pct_Used", "Virtual_Storage_Pct_Avail", "Total_Swap_Space_MB", "Used_Swap_Space_MB", "Avail_Swap_Space_MB", "Used_Swap_Space_Pct", "Avail_Swap_Space_Pct", "Total_Real_Mem_MB", "Used_Real_Mem_MB", "Avail_Real_Mem_MB", "Used_Real_Mem_Pct", "Avail_Real_Mem_Pct", "Page_Faults", "Page_Reclaims", "Page_Ins", "Page_Outs", "Page_In_Reqs", "Page_Out_Reqs", "Page_In_KB_S", "Page_Out_KB_S", "Page_In_1Min", "Page_In_5Min", "Page_In_15Min", "Page_In_60Min", "Page_Out_1Min", "Page_Out_5Min", "Page_Out_15Min", "Page_Out_60Min", "Page_Scan", "Page_Scan_KB", "Page_Scan_1Min", "Page_Scan_5Min", "Page_Scan_15Min", "Page_Scan_60Min", "ARC_Size_MB", "Net_Memory_Used", "Net_Memory_Avail", "Net_Memory_Used_Pct", "Net_Memory_Avail_Pct", "Non_Comp_Memory", "Comp_Memory", "Decay_Rate", "Repaging_Rate", "Pages_Read_per_Sec", "Pages_Written_per_Sec", "Paging_Space_Free_Pct", "Paging_Space_Used_Pct", "Paging_Space_Read_per_Sec", "Paging_Space_Write_per_Sec", "Comp_Mem_Pct", "Non_Comp_Mem_Pct", "Filesys_Avail_Mem_Pct", "MEMAV_WHSC", "RMAVA_WHSC", "RMUSD_WHSC", "MEMUS_WHSC", "Non_Comp_Mem_MB", "Comp_Mem_MB", "Filesys_Avail_Mem_MB", "Process_Mem_Pct", "PRMEP_WHSC", "System_Mem_Pct", "SYMEP_WHSC", "File_Repl_Mem_Pct", "FRMEP_WHSC", "File_Repl_Min_Mem_Pct", "FRMIP_WHSC", "File_Repl_Max_Mem_Pct", "FRMAP_WHSC", "Unlocked_Pageable_Memory", "UPM_WHSC", "Scanned_Pages", "SP_WHSC", "System_Paging", "SYSP_WHSC", "Total_Disk_Swap_Space_MB", "TDSS_WHSC", "Used_Disk_Swap_Space_MB", "UDSS_WHSC", "Avail_Disk_Swap_Space_MB", "ADSS_WHSC", "Used_Disk_Swap_Space_Pct", "UDSSP_WHSC", "Avail_Disk_Swap_Space_Pct", "ADSSP_WHSC", "Real_Mem_on_Disk_Swap_Ratio", "MR_WHSC", "Page_Size", "PSIZE_WHSC"] convert => { "TMZDIFF" => "integer" "Total_Virtual_Storage_MB" => "integer" "Used_Virtual_Storage_MB" => "integer" }
}

mutate {
convert => {"WRITETIME" => "integer"}
}
mutate {
convert => {"Timestamp" => "integer"}
}
date{ match => [ "WRITETIME", "UNIX_MS" ]
target => "WRITETIME"
}
date{ match => [ "Timestamp", "UNIX_MS" ]
target => "Timestamp"
}
}
output {
elasticsearch {
hosts => "localhost"
index => "mem-index2"
template_overwrite => true
}
stdout {
codec => rubydebug
}

Issues :

  1. I don't see time stamp extracting. Below two
    Timestamp
    WRITETIME

  2. Have also tried with out mutate, without converting to integer

Thanks for the support in advance!

2

UNIX_MS is milliseconds since epoch. Your timestamps are microseconds since epoch, so instead of being about 40 years past the epoch (1970) your timestamp is 40,000 years past the epoch: "Timestamp" => 39078-04-11T00:48:23000Z. Try

ruby { code => "event.set('WRITETIME', event.get('WRITETIME') / 1000)" }

and you will get "WRITETIME" => 2007-02-09T07:43:20.040Z

3

Cool, it worked.
Now other issue is not able to get this viewed in KIBANA as these are from 2007.
So is there a way that we can see in KIBANA ?

4

Now other issue is not able to get this viewed in KIBANA as these are from 2007.
So is there a way that we can see in KIBANA ?

Kibana doesn't care if you have old logs. Are the indexes being created in the first place?

5

hi,

yup it did got created.

Thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.