Unable to preserve value in global variable

Hello,

After searching and surfing for hours and hours, I am still unable to preserve the value of an event field in a global variable.
This is what I am trying to achive.
I have a log file something like:

INFO: ServerInstanceId : ServerManager_1 : starting scan for target id : 1 and walk : 1 of scan mode : single server
FINEST: com.soft.filetemp.DataFetcher FileUpload File: l:\test10\ntlm1.pcap DownloadByteContentSize: 1232 Time taken: 3

My logstash config file:

filter {
  grok {

    match => { "message" => "(?<Custom>[Ff]inest|FINEST|[Ff]iner|FINER|[Ff]ine|FINE|[Dd]ebug|DEBUG|[Ss]evere|SEVERE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Ff]atal|FATAL?):\sServerInstanceId\s:\s%{DATA:ServerInstanceId}\s:\s%{DATA:ScanState}\sfor\s%{DATA}:\s%{NUMBER:TargetID}\sand\s%{DATA}:\s%{NUMBER:WalkID}\s%{DATA}:\s%{GREEDYDATA:ScanMode}" }
    
    match => { "message" => "(?<Custom>[Ff]inest|FINEST|[Ff]iner|FINER|[Ff]ine|FINE|[Dd]ebug|DEBUG|[Ss]evere|SEVERE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Ff]atal|FATAL?):\s%{JAVACLASS:JavaClass}\s(?<DetectionAction>[Ff]ile[Dd]ownload|FILEDOWNLOAD?)\sFile:\s%{GREEDYDATA:FilePath}DownloadByteContentSize:\s%{NUMBER:DownloadByteContentSize}\s%{GREEDYDATA:FilePath}:\s%{NUMBER:TimeTaken}" }
  }
mutate {
        add_field => {"ScanIDConfig" => "%{@timestamp}%{ServerInstanceId}%{TargetID}%{WalkID}"}
  }
ruby {
        init => "@@ScanIDConfig = ''"
        code => "@@ScanIDConfig=event.get('[ScanIDConfig]')
        event.set('ScanIDConfig',@@ScanIDConfig)"
  }
}

Based on first log line, I am generating a custom field named ScanIDConfig. I want to store and use the value of this variable in all subsequent events.
However, logstash output for second event depicts:

"ScanIDConfig" => "2019-03-28T09:57:40.680Z%{ServerInstanceId}%{TargetID}%{WalkID}"

For first event, it is as desired:

"ScanIDConfig" => "2019-03-28T10:19:51.160Z ServerManager_111",

I tried multiple combinations of Ruby filter, but no luck.

Can someone please help me here.

I think you want something more like

ruby {
    init => "@@ScanIDConfig = ''" 
    code => "
        if @@ScanIDConfig == ''
            @@ScanIDConfig=event.get('[ScanIDConfig]') 
        else
            event.set('ScanIDConfig',@@ScanIDConfig)" }
        end
    "
}

Generally, using ruby variables is a bad idea. You are making assumptions about event ordering that logstash does not guarantee and it limits you to a single worker thread.

Thank you Badger for your response.
I used the same configuration, however, it did not work. Any clue?
I am still getting variables without their values.

"ScanIDConfig" => "2019-03-28T13:56:07.751Z%{ServerInstanceId}%{TargetID}%{WalkID}",

Another option would be

ruby {
    code => "
        f = event.get('[ScanIDConfig]')
        if f
            @@ScanIDConfig = f 
        else
            event.set('ScanIDConfig',@@ScanIDConfig)" }
        end
    "
 }

Appreciate your response Badger!
In ruby section, there seems an issue with the assignment of custom field (ScanIDConfig) created using mutate plugin. Unsure why.
If I use any normal data field (not custom one e.g. ServerInstanceId) , in ruby block, the snippet provided by you works as desired.

ruby {
    init => "@@ServerInstanceId = ''"
    code => "
        if @@ServerInstanceId == ''
            @@ServerInstanceId=event.get('[ServerInstanceId]') 
        else
            event.set('ServerInstanceId',@@ServerInstanceId)
        end
    "
}

Anyway, if there is any better way to achieve my purpose which can allow me to get rid of the worker thread restriction.

No, logstash isn't really designed to support this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.