Replace @timestamp with timestamp in my message - logstash

Elastic Stack Logstash
1

Hi Everyone

i need to replace @timestamp with the timestamp from log file(message). still i can't mention complete message format. i can use like starts with timestamp. i tried below code

grok {
match => { "message" => ["^{timestamp}"] }
}

     date {
    		match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    		target => "@timestamp"
    	}
     }

This is the output i got

{"tags":["_grokparsefailure"],"message":["<158>2019-01-07 05:00:31 UTC","User.Info","192.156.15.13","\"Jan 6 22:26:55 PA-200.tdc.crsg 1","2019/01/06 22:26:54","001606074152","TRAFFIC","end","1","2019/01/06 22:26:54","145.123.15.23","10.2.2.3","117.6.162.63","192.156.15.13","WAN-LAN","","","ms-rdp","vsys1","untrust","trust","ethernet1/3","ethernet1/4","Kiwi Syslog","2019/01/06 22:26:54","64750","1","57890","3389","57890","3389","0x400050","tcp","allow","3705","1708","1997","18","2019/01/06 22:26:35","4","any","0","361598833","0x0","VN","IN","0","10","8","tcp-rst-from-client","0","0","0","0","","PA-200","from-policy\" "],"@timestamp":"2019-01-28T12:29:34.379Z"}

The output i required is

{"tags":["_grokparsefailure"],"message":["<158>2019-01-07 05:00:31 UTC","User.Info","192.156.15.13","\"Jan 6 22:26:55 PA-200.tdc.crsg 1","2019/01/06 22:26:54","001606074152","TRAFFIC","end","1","2019/01/06 22:26:54","145.123.15.23","10.2.2.3","117.6.162.63","192.156.15.13","WAN-LAN","","","ms-rdp","vsys1","untrust","trust","ethernet1/3","ethernet1/4","Kiwi Syslog","2019/01/06 22:26:54","64750","1","57890","3389","57890","3389","0x400050","tcp","allow","3705","1708","1997","18","2019/01/06 22:26:35","4","any","0","361598833","0x0","VN","IN","0","10","8","tcp-rst-from-client","0","0","0","0","","PA-200","from-policy\" "],"@timestamp":"2019-01-07 05:00:31 UTC"}

can anyone help me regrading this pls :frowning_face:

Thanks.

2

Your message does not start with timestamp, it starts with <158>. And the time format is wrong. Your message contains "YYYY-MM-dd HH:mm:ss ZZZ"

3

i don't understand Mr.Badger that y iam getting this number

4

I guess it is a syslog PRI field.

1 Like
5

can't we remove that?

6

Hi,

Usually when you see messages starting with <158> or similar numbers the input is not quite correct. Can you share your logstash config and your filters?

Edit: I am guessing that you are sending syslog messages but your input is a tcp / udp input filter. When you change the input filter to syslog, you should not see the <158> anymore

Paul.

7

hi pjanzen,

This is my conf file

input {
            file{
        	path => "E:\softwares\logstash-6.5.4\bin\PA_FW_Traffic-2018-09-10.txt"
          start_position => beginning
         }
         }
        filter{
        mutate { 
        	remove_field => ["facility_label","facility","@version","priority","tags","host","timestamp","severity_label","severity","logsource"]
        	split => { "message" => "," }
         }
        grok {
            match => { "message" => ["^{timestamp}"] }
        	}

         date {
        		match => [ "timestamp", "yyyy-MM-dd :HH:mm:ss ZZZ" ]
        		target => "@timestamp"
        	}
         }
        output {
            stdout {}
            file {
                path => "E:/testmessage.txt"
         codec => line { format => ["%{@timestamp},%{message}"] } 	
        	}
        }
8

Can you also share some input from the txt file?

Also the remove_field part you have are all syslog field, normally you get those when you send data over the network..

Paul.

9

2019-01-07 05:00:31 UTC,User.Info,192.168.11.1,"Jan 6 22:26:55 PA-200.tdc.crsg 1,2019/01/06 22:26:54,001606074152,TRAFFIC,end,1,2019/01/06 22:26:54,10.0.26.5,123.69.8.19,153.26.16.3,186.23.56.5,WAN-LAN,,,ms-rdp,vsys1,untrust,trust,ethernet1/3,ethernet1/4,Kiwi Syslog,2019/01/06 22:26:54,64750,1,57890,3389,57890,3389,0x400050,tcp,allow,3705,1708,1997,18,2019/01/06 22:26:35,4,any,0,361598833,0x0,VN,IN,0,10,8,tcp-rst-from-client,0,0,0,0,,PA-200,from-policy"

10

Where the <158> is coming from in this example I do not know.
You can update your grok pattern and match for this %{TIMESTAMP_ISO8601:timestamp} that will fill the timestamp variable.

So instead of this:

grok {
   match => { "message" => ["^{timestamp}"] }
}

You get this:

grok {
   match => { "message" => ["^%{TIMESTAMP_ISO8601:timestamp}"] }
}

Then your date match can look like this.

date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
    target => "@timestamp"
}

Hope this helps.

Paul.

1 Like
11

I created a working config based on your information.
Logstash version 6.5.4

I removed the mutate line as that does not do anything anyway.

Filter:

input {
    file{
        path => "/home/pjanzen/test_input.txt"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:replace_timestamp}" }
    }
    date {
      match => ['replace_timestamp', 'yyyy-MM-dd HH:mm:ss']
      timezone => "UTC"
      target => "@timestamp"
    }
}
output {
  stdout {
    codec => rubydebug
  }
}

Ouput:
Before timestamp match:

{
    "@timestamp" => 2019-01-28T15:18:52.458Z,
       "message" => "2019-01-07 05:00:31 UTC,User.Info,192.168.11.1,\"Jan 6 22:26:55 PA-200.tdc.crsg 1,2019/01/06 22:26:54,001606074152,TRAFFIC,end,1,2019/01/06 22:26:54,10.0.26.5,123.69.8.19,153.26.16.3,186.23.56.5,WAN-LAN,,,ms-rdp,vsys1,untrust,trust,ethernet1/3,ethernet1/4,Kiwi Syslog,2019/01/06 22:26:54,64750,1,57890,3389,57890,3389,0x400050,tcp,allow,3705,1708,1997,18,2019/01/06 22:26:35,4,any,0,361598833,0x0,VN,IN,0,10,8,tcp-rst-from-client,0,0,0,0,,PA-200,from-policy\"",
     "timestamp" => "2019-01-07 05:00:31",
          "host" => "tb-clog-ls1",
      "@version" => "1",
          "path" => "/home/pjanzen/test_input.txt"
}

After timestamp match.

{
           "@timestamp" => 2019-01-07T05:00:31.000Z,
             "@version" => "1",
                 "host" => "tb-clog-ls1",
              "message" => "2019-01-07 05:00:31 UTC,User.Info,192.168.11.1,\"Jan 6 22:26:55 PA-200.tdc.crsg 1,2019/01/06 22:26:54,001606074152,TRAFFIC,end,1,2019/01/06 22:26:54,10.0.26.5,123.69.8.19,153.26.16.3,186.23.56.5,WAN-LAN,,,ms-rdp,vsys1,untrust,trust,ethernet1/3,ethernet1/4,Kiwi Syslog,2019/01/06 22:26:54,64750,1,57890,3389,57890,3389,0x400050,tcp,allow,3705,1708,1997,18,2019/01/06 22:26:35,4,any,0,361598833,0x0,VN,IN,0,10,8,tcp-rst-from-client,0,0,0,0,,PA-200,from-policy\"",
    "replace_timestamp" => "2019-01-07 05:00:31",
                 "path" => "/home/pjanzen/test_input.txt"
}

As you can see the timestamp in your event overwrites the timestamp for the document.

12

Thank u Mr.Pjanzen

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.