Regarding timestamp index from log file name

[Kalyan_MB]

Hi,

Problem: need to read time stamp from the input file name (filebeat ) from logstash filter.

input file name name format:
-rw-r--r-- 1 root root 204 Apr 3 15:22 free_data_09_09_13_20.txt
-rw-r--r-- 1 root root 204 Apr 3 15:22 free_data_09_09_13_10.txt
-rw-r--r-- 1 root root 204 Apr 3 15:22 free_data_09_09_13_00.txt

------ free_data_DD_MM_hh_mm (date_Month_hour_min)

now i want to read this time stamp from the input file name and create an index along with file content, which i will use later to plot graph against time stamp vs file-data-values.

[Badger]

Does this help?

    grok { match => { "someField" => "free_data_(?<ts>[0-9_]+).txt" } }
    date { match => [ "ts", "dd_MM_HH_mm" ] }

You do not have a year in your timestamp, so the date filter will guess, and sometimes you will not like its guess. There are issues for that here and here.

[Kalyan_MB]

facing this problem..
2019-04-03T20:42:54.626+0530 INFO log/harvester.go:216 Harvester started for file: /home/vankata/190_APS_QUALIFICATION/kalyan_elk_logs/free_data_11_09_02_30.txt
2019-04-03T20:42:54.626+0530 INFO prospector/prospector.go:121 Prospector ticker stopped
2019-04-03T20:42:54.626+0530 INFO log/prospector.go:411 Scan aborted because prospector stopped.
2019-04-03T20:42:54.626+0530 INFO prospector/prospector.go:121 Prospector ticker stopped
2019-04-03T20:42:54.626+0530 INFO prospector/prospector.go:138 Stopping Prospector: 8728499415371259904
2019-04-03T20:42:54.626+0530 INFO prospector/prospector.go:121 Prospector ticker stopped
2019-04-03T20:42:54.626+0530 INFO log/harvester.go:216 Harvester started for file: /home/vankata/190_APS_QUALIFICATION/kalyan_elk_logs/free_data_11_09_10_40.txt

filebeat version: filebeat-6.2.4

Note: tried to delete registry files by using clean_* command couldn't succeed though

[Badger]

That is a filebeat question and you should ask in that forum.

[Kalyan_MB]

ah yes!, this is logstash forum. thanks @Badger

[Kalyan_MB]

@Badger, used the filter in logstash.conf as below looks like its not solving problem.

---

input {
beats {
port => 5044
}
}

filter {

grok { match => { "someField" => "free_data_(?[0-9_]+).txt" } }
date { match => [ "ts", "dd_MM_HH_mm" ] }
if [message] =~ /Mem:/ {
dissect { mapping => {'message' => '%{mem_type->} %{total_mem->} %{used_mem->} %{free_mem->} %{shared_mem->} %{cache_mem->} %{availablemem}'} }
} else if [message] =~ /Swap:/ {
dissect { mapping => {'message' => '%{mem_type->} %{total_mem->} %{used_mem->} %{free_mem->}'} }
}

}

output {
if [service_name] == "cfx" {
elasticsearch
{
#path => "/var/log/sdl_logs/%{vnf_id}/%{vm_type}/%{instance_id}/%{service_name}/%{app_name}_%{+yyyy-MM-dd-HH}.log"
#codec => line { format => "%{message}" }
#gzip => true
hosts => ["http://x.x.x.x:9200"]
index => "freedata"

---

because after index pattern i cant see timestamp(ts) in discover page