Logstash Custom Timestamp issue

Elastic Stack Logstash
1

Hi,

I am trying to put transaction data into elasticsearch, i want to override @timestamp with actual transaction time, but after converting timestamp , getting different time.

below if my conf file:

input {
  file{
	path => "/home/optimus/kibana/data/TX_DUMP_PPBL2019-03-31_16:41:431554030703.csv"
	start_position =>"beginning"
	sincedb_path => "/dev/null"
  }
}

filter{
  csv{
  separator => ","
  skip_header => true
  columns => [ "Tid", "TxTime", "Amount", "BankName", "IFSC","Account" ,"Sender Name","Sender Cell","RRN","BankTid","Status","Status Desc","ResCode","ResDesc","Pipe","Recon","ReqAt","ResAt","ResTime" ]

  }

  mutate {
	convert => {
		"Amount" => "float"
		"Status" => "integer"
		"Pipe" => "integer"
		"Recon" => "integer"
		"TxTime" => "string"
  }
}

  date {
	  match => [ "TxTime" , "ISO8601" , "yyyy-MM-dd HH:mm:ss" ]
	  target => "@timestamp"
	  remove_field => [ "TxTime", "timestamp remove" ]
  }

  mutate{
	add_field => {
	"Tag" => "Test"
	}

	remove_field => [ "host", "@version","path"]
	rename => {

	}

  }
}

output{
  stdout{
  codec => rubydebug
  }
}


Sample input: 
1300112054,2019-03-25 13:47:23,5000,SBI,SBIN0000001,63026818178,Rohit,8851929333,NA,1000007219,0,Fail,1105,Corporate First leg of transactions failed or pending,9,0,2019-03-25 13:47:23,2019-03-25 13:47:24,1

Output:
{
	"Status Desc" => "Fail",
		  "Recon" => 0,
		"BankTid" => "1000007219",
	 "@timestamp" => **2019-03-25T08:17:23.000Z**,
			"Tid" => "1300112054",
		"ResTime" => "1",
		 "Amount" => 5000.0,
	"Sender Name" => "Rohit",
	   "BankName" => "SBI",
		  "ReqAt" => "2019-03-25 13:47:23",
			"Tag" => "Test",
	"Sender Cell" => "8851929333",
		"ResDesc" => "Corporate First leg of transactions failed or pending",
		"Account" => "63026818178",
		"ResCode" => "1105",
			"RRN" => "NA",
		"message" => "1300112054,**2019-03-25 13:47:23**,5000,SBI,SBIN0000001,63026818178,Rohit,8851929333,NA,1000007219,0,Fail,1105,Corporate First leg of transactions failed or pending,9,0,2019-03-25 13:47:23,2019-03-25 13:47:24,1",
		  "ResAt" => "2019-03-25 13:47:24",
		   "IFSC" => "SBIN0000001",
		 "Status" => 0,
		   "Pipe" => 9
}

Can someone please help why there is a different of around 5 hrs, and how i can correct this?

2

logstash is assuming that the timestamp in the log is in your local timezone and it is converting it to UTC. It is doing that by subtracting 5:30, which suggests you are in the Asia/Kolkota timezone.

The Elastic stack always stores times as UTC. The timestamp in the logfile is in some other timezone, then pass the timezone option to the date filter. If the timestamp in the log file is in Asia/Kolkota then you do not need to change anything -- it is working correctly.

3

Thanks

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.