Unable to preserve value in global variable

Elastic Stack Logstash
1

Hello,

After searching and surfing for hours and hours, I am still unable to preserve the value of an event field in a global variable.
This is what I am trying to achive.
I have a log file something like:

INFO: ServerInstanceId : ServerManager_1 : starting scan for target id : 1 and walk : 1 of scan mode : single server
FINEST: com.soft.filetemp.DataFetcher FileUpload File: l:\test10\ntlm1.pcap DownloadByteContentSize: 1232 Time taken: 3

My logstash config file:

filter {
  grok {

    match => { "message" => "(?<Custom>[Ff]inest|FINEST|[Ff]iner|FINER|[Ff]ine|FINE|[Dd]ebug|DEBUG|[Ss]evere|SEVERE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Ff]atal|FATAL?):\sServerInstanceId\s:\s%{DATA:ServerInstanceId}\s:\s%{DATA:ScanState}\sfor\s%{DATA}:\s%{NUMBER:TargetID}\sand\s%{DATA}:\s%{NUMBER:WalkID}\s%{DATA}:\s%{GREEDYDATA:ScanMode}" }
    
    match => { "message" => "(?<Custom>[Ff]inest|FINEST|[Ff]iner|FINER|[Ff]ine|FINE|[Dd]ebug|DEBUG|[Ss]evere|SEVERE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Ff]atal|FATAL?):\s%{JAVACLASS:JavaClass}\s(?<DetectionAction>[Ff]ile[Dd]ownload|FILEDOWNLOAD?)\sFile:\s%{GREEDYDATA:FilePath}DownloadByteContentSize:\s%{NUMBER:DownloadByteContentSize}\s%{GREEDYDATA:FilePath}:\s%{NUMBER:TimeTaken}" }
  }
mutate {
        add_field => {"ScanIDConfig" => "%{@timestamp}%{ServerInstanceId}%{TargetID}%{WalkID}"}
  }
ruby {
        init => "@@ScanIDConfig = ''"
        code => "@@ScanIDConfig=event.get('[ScanIDConfig]')
        event.set('ScanIDConfig',@@ScanIDConfig)"
  }
}

Based on first log line, I am generating a custom field named ScanIDConfig. I want to store and use the value of this variable in all subsequent events.
However, logstash output for second event depicts:

"ScanIDConfig" => "2019-03-28T09:57:40.680Z%{ServerInstanceId}%{TargetID}%{WalkID}"

For first event, it is as desired:

"ScanIDConfig" => "2019-03-28T10:19:51.160Z ServerManager_111",

I tried multiple combinations of Ruby filter, but no luck.

Can someone please help me here.

2

I think you want something more like

ruby {
    init => "@@ScanIDConfig = ''" 
    code => "
        if @@ScanIDConfig == ''
            @@ScanIDConfig=event.get('[ScanIDConfig]') 
        else
            event.set('ScanIDConfig',@@ScanIDConfig)" }
        end
    "
}

Generally, using ruby variables is a bad idea. You are making assumptions about event ordering that logstash does not guarantee and it limits you to a single worker thread.

3

Thank you Badger for your response.
I used the same configuration, however, it did not work. Any clue?
I am still getting variables without their values.

"ScanIDConfig" => "2019-03-28T13:56:07.751Z%{ServerInstanceId}%{TargetID}%{WalkID}",

4

Another option would be

ruby {
    code => "
        f = event.get('[ScanIDConfig]')
        if f
            @@ScanIDConfig = f 
        else
            event.set('ScanIDConfig',@@ScanIDConfig)" }
        end
    "
 }
5

Appreciate your response Badger!
In ruby section, there seems an issue with the assignment of custom field (ScanIDConfig) created using mutate plugin. Unsure why.
If I use any normal data field (not custom one e.g. ServerInstanceId) , in ruby block, the snippet provided by you works as desired.

ruby {
    init => "@@ServerInstanceId = ''"
    code => "
        if @@ServerInstanceId == ''
            @@ServerInstanceId=event.get('[ServerInstanceId]') 
        else
            event.set('ServerInstanceId',@@ServerInstanceId)
        end
    "
}

Anyway, if there is any better way to achieve my purpose which can allow me to get rid of the worker thread restriction.

6

No, logstash isn't really designed to support this.