Hi, I just created a pipeline and transfer .csv log file to index. I want to find the started time and calculate the duration time. I used the code from official document:
filter {
csv {
separator => ","
columns => [ "Timestamp", "SeverityName", "EventName", "EntityName", "Application", "Operation", "TransactionId", "SessionId", "OperationId", "PartyName", "ClientId", "Host", "LogId" ]
}
mutate {convert => ["SeverityName", "string"]}
mutate {convert => ["EventName", "string"]}
mutate {convert => ["EntityName", "string"]}
mutate {convert => ["Application", "string"]}
mutate {convert => ["Operation", "string"]}
mutate {convert => ["TransactionId", "string"]}
mutate {convert => ["SessionId", "string"]}
mutate {convert => ["OperationId", "string"]}
mutate {convert => ["PartyName", "string"]}
mutate {convert => ["ClientId", "string"]}
mutate {convert => ["Host", "string"]}
mutate {convert => ["LogId", "integer"]}
date {
match => ["Timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
target => "@timestamp"
remove_field => ["Timestamp"]
}
if [EventName] == "Sending" and [EntityName] == "Response" {
elasticsearch {
index => "prodlogssmall_10"
query => "EventName:Received AND EntityName:Request AND SessionId:%{[SessionId]}"
fields => { "@timestamp" => "started" }
}
date {
match => ["[started]", "YYYY-MM-dd HH:mm:ss.SSS"]
target => "[started]"
}
ruby {
code => "
event.set('duration', (event.get('@timestamp') - event.get('started')) * 1000)
"
}
}
When I run logstash, the started field can be created successfully and I can see it in ES. But duration filed will be failed to creat because it seems there is no filed of 'started' when it is calculated. The error message is:
[2019-07-05T13:32:21,186][ERROR][logstash.filters.ruby ] Ruby exception occurred: can't convert nil into an exact number
I am confused that my code is almost same as the code from official document:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html
Could you please help me? Many thanks.