Extract date from filename, time from log line

Elastic Stack Elasticsearch
1

Hi
on logstash need to use file as input, output as http.
here is the string must be send:
mymeasure,tag=mytag field="myfield" 1689682934

this part "1689682934" is timestamp.

now question is how can i extract date from filename, time from log line and create epoch time format like 1689682934 and put in end of line.

here is file name:
/tmp/log.cus.20230720

here is log line:
00:00:13.441 cus module: A[srv1]B[551]C[000]

expected timestamp
20230720 00:00:13.441

expected timestamp in epochtime format
1689682934

Any idea?
Thanks

2 
input {
  file {
    path => "/tmp/log.cus.*"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

filter {
  grok {
    match => { "message" => "%{TIME:time} cus module: %{GREEDYDATA}" }
  }

  date {
    match => [ "time", "HH:mm:ss.SSS" ]
    target => "timestamp"
    locale => "en"
  }

  ruby {
    code => '
      require "date"
      filename_date = File.basename(event.get("path")).scan(/\d+/)[0]
      log_time = event.get("timestamp").to_datetime.strftime("%Y%m%d")
      combined_time = "#{filename_date} #{log_time} #{event.get("timestamp")}"
      epoch_time = DateTime.parse(combined_time).to_time.to_i
      event.set("epoch_time", epoch_time)
    '
  }

  mutate {
    remove_field => [ "path", "time", "timestamp" ]
  }
}

output {
  http {
    url => "http://your-api-endpoint"
    http_method => "post"
    headers => { "Content-Type" => "application/json" }
    format => "json"
    message_key => "message"
  }
}
  • Reas data from file. Codec treats each line as json object
  • grok extracts time and stores in time
  • date filter converts it logstash timestamp
  • ruby converts to epoch_time
  • mutate filters unnecessary fields like timestamp
  • http sends it with post
3

@PodarcisMuralis Hi
I try config that you mentioned but still consider current timestamp on stdout not log file timestamp!

4

@PodarcisMuralis

here is the stdout:
"@timestamp" => 2023-08-12T05:53:50.285594827Z,

expected result:
"@timestamp" => 2023-07-20T00:00:13.441Z,

logfilename: log.cus0.20230720
logline: 00:00:13.441 cus module: A[srv1]B[551]C[000]

here is the error:

[ERROR] 2023-08-12 09:40:54.738 [[main]>worker3] ruby - Ruby exception occurred: no implicit conversion of nil into String {:class=>"TypeError", :backtrace=>["org/jruby/RubyFile.java:523:in basename'", "(ruby filter code):4:in block in filter_method'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in inline_script'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in block in multi_filter'", "org/jruby/RubyArray.java:1865:in each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:301:in block in start_workers'"]}`

Any idea?
Thanks

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.