Converting hexadecimal input into timestamp

Elastic Stack Logstash
1

I have a qmail server logs where the timestamp is given in different hexadecimal format how can i concert this again in timestamp????
@400000005cc363f11d3becc4 sslserver: pid 4983 from 209.85.210.174

2

I've searched it's in TAI64N format..can i get some help on how can i convert it into DD:MM:YYYY format??

3

The date filter can handle TAI64N

    mutate { add_field => { "someField" => "400000005cc363f11d3becc4" } }
    date { match => [ "someField", "TAI64N" ] }

will get you

"@timestamp" => 2019-04-26T20:02:47.490Z,

If you want to format @timestamp into another field with a particular format you can search the forum for examples using strftime in a ruby filter.

4

I've written this grok for my qmail smtp logs where I am trying to segregate the logs. I am getting error in my grok since I am sure that I am mistaking in formatting please help me:
input {
file {
path => "/elk/MAILGW-SMPTPD/*"
start_position => "beginning"
}
}
filter
{

    grok
        { match => { "message" => "@%{S3_REQUEST_LINE}%{S3_REQUEST_LINE}%{NUMBER:process_id}%{S3_REQUEST_LINE}%{WORD:msg}%{S3_REQUEST_LINE}%{IPV4:requesting_ip}%{NOTSPACE} %{NOTSPACE}%{S3_REQUEST_LINE}:%{JAVALOGMESSAGE:sender} %{NOTSPACE}:%{JAVALOGMESSAGE:receiver}" }
        }
        if "_grokparsefailure" in [tags] {
        grok
              {
                  match=> {
                            "message"=>"@%{S3_REQUEST_LINE}%{S3_REQUEST_LINE}%{NUMBER:process_id}%{S3_REQUEST_LINE}%{IPV4:ip}"
                          }
                  mutate { add_field => { "timestamp" => "verb" } }
               date { match => [ "timestamp", "TAI64N" ] }
              }
    }



geoip { source => "requesting_ip" }

}

  output
    {
    elasticsearch {
          hosts => ["10.11.109.7:9200"]
            index => "qmail_smtpdlogs"
                  }
     }
5 
sample of my log file is:
    @400000005cc363e01c89139c sslserver: dhparam 2846 /package/host/superscript.com/net/ucspi-ssl/etc/dh1024.pem 2048
    @400000005cc363e01c89139c sslserver: status: 0/40
    @400000005cc363f033f45c94 sslserver: status: 1/40
    @400000005cc363f11d3becc4 sslserver: pid 4983 from 209.85.210.174
    @400000005cc363f11d3bf494 sslserver: ok 4983 MailGW-1.rrcat.gov.in:192.168.200.25:25 mail-pf1-f174.google.com:209.85.210.174::39538
    @400000005cc363f325362944 qmail-smtpd: pid 4983 Accept::SPF::Rcpthosts_Rcptto P:ESMTP S:209.85.210.174:mail-pf1-f174.google.com H:mail-pf1-f174.google.com F:gbsubramani@ipgi.co.in T:neel@rrcat.gov.in
    @400000005cc363f336635cdc ERROR: Could not connect to clamd on 127.0.0.1: Connection refused
    @400000005cc363f408c6d574 sslserver: tls 4983 accept
    @400000005cc363f408c844a4 sslserver: ended by 4912 status 0
    @400000005cc363f408c8488c sslserver: status: 0/40
    @400000005cc364181f388bcc sslserver: status: 1/40
    @400000005cc364190b5db0dc sslserver: pid 8473 from 54.240.10.44
    @400000005cc364190b5db8ac sslserver: ok 8473 MailGW-1.rrcat.gov.in:192.168.200.25:25 a10-44.smtp-out.amazonses.com:54.240.10.44::34998
    @400000005cc3641a2b6f0e84 qmail-smtpd: pid 8473 Accept::SPF::Rcpthosts_Rcptto P:ESMTP S:54.240.10.44:a10-44.smtp-out.amazonses.com H:a10-44.smtp-out.amazonses.com F:0100016a5981e3e4-5bbce702-15fe-42b6-b6aa-eae2dc6d069d-000000@amazonses.com T:gramu@rrcat.gov.in
    @400000005cc3641b08e0f8dc qmail-smtpd: pid 8473 Accept::SPF::Rcpthosts_Rcptto P:ESMTP S:54.240.10.44:a10-44.smtp-out.amazonses.com H:a10-44.smtp-out.amazonses.com F:0100016a5981e3e4-5bbce702-15fe-42b6-b6aa-eae2dc6d069d-000000@amazonses.com T:rnchaudhary@rrcat.gov.in
    @400000005cc3641f3466a664 sslserver: status: 2/40
    @400000005cc364201a96b594 sslserver: pid 11772 from 54.240.11.8
6

I would grok the common prefix then separately parse the restOfLine

    grok { match => { "message" => "@(?<[@metadata][ts]>[a-z0-9]{24}) (?<daemon>[^:]+): %{GREEDYDATA:restOfLine}" } }
    date { match => [ "[@metadata][ts]", "TAI64N" ] }
7

I tried using common prefix but I ain't expert in writing groks so I tried that way..mmm...is there any specific mechanism through which I can learn writing the groks???

8

I like using this Grok Debugger to discover and test patterns: https://grokdebug.herokuapp.com/

9

I did tried that website for searching patterns

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.