Duration Time Calculation: missing started time

Hi, I just created a pipeline and transfer .csv log file to index. I want to find the started time and calculate the duration time. I used the code from official document:
filter {
csv {
separator => ","
columns => [ "Timestamp", "SeverityName", "EventName", "EntityName", "Application", "Operation", "TransactionId", "SessionId", "OperationId", "PartyName", "ClientId", "Host", "LogId" ]
}

	mutate {convert => ["SeverityName", "string"]}
	mutate {convert => ["EventName", "string"]}
	mutate {convert => ["EntityName", "string"]}
	mutate {convert => ["Application", "string"]}
	mutate {convert => ["Operation", "string"]}
	mutate {convert => ["TransactionId", "string"]}
	mutate {convert => ["SessionId", "string"]}
	mutate {convert => ["OperationId", "string"]}
	mutate {convert => ["PartyName", "string"]}
	mutate {convert => ["ClientId", "string"]}
	mutate {convert => ["Host", "string"]}
	mutate {convert => ["LogId", "integer"]}
		
	date {
		  match => ["Timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
		  target => "@timestamp"
		  remove_field => ["Timestamp"]
	}
		
	if [EventName] == "Sending" and [EntityName] == "Response" {
	    
		elasticsearch {
		    index => "prodlogssmall_10"
			query => "EventName:Received AND EntityName:Request AND SessionId:%{[SessionId]}"
			fields => { "@timestamp" => "started" }
		}
		 
		date {
			  match => ["[started]", "YYYY-MM-dd HH:mm:ss.SSS"]
			  target => "[started]"
		}			 	
    
	    ruby {
			  code => "
			        event.set('duration', (event.get('@timestamp') - event.get('started')) * 1000) 				
					"
		}	
	}		

When I run logstash, the started field can be created successfully and I can see it in ES. But duration filed will be failed to creat because it seems there is no filed of 'started' when it is calculated. The error message is:
[2019-07-05T13:32:21,186][ERROR][logstash.filters.ruby ] Ruby exception occurred: can't convert nil into an exact number

I am confused that my code is almost same as the code from official document:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

Could you please help me? Many thanks.

I found the problem is that I have to set batch size as 1. Hence, the elasticsearch filter plugin can find the started time of a session in ES. However, it is very slow if the size cannot be set larger. Is there any other solution to calculate duration time?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.