[SOLVED] Logstash 5.4 Grok match partially working, can't figure why

Elastic Stack Logstash
1

Greetings,

For my first thread to you, I would like to submit an issue to your expertise.

I have been using ELK stack (5.4) to gather data from our local radio station stream.
I set up a Logstash file input which looks like this

...
12.255.255.187#Bose Monitoring Service#113369#5136220
...

I have been using successfully the grok parser as follow:

filter                                                                                                                   
{                                                                                                                        
    grok {                                                                                                               
            match => [ 
                 "message", "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}"                                  
            ]                                                                                                            
         }                                                                                                               
        }                                                                                                                
                                                                                                                         
        geoip {                                                                                                          
                source => "IP"                                                                                           
                target => "geoip"                                                                                        
                add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]                                         
                add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]                                         
        }                                                                                                                
                                                                                                                         
        mutate {                                                                                                         
                convert => [ "[geoip][coordinates]", "float"]                                                            
        }                                                                                                                
                                                                                                                         
        useragent {                                                                                                      
                source => "UserAgent"                                                                                    
                prefix=> "browser"                                                                                       
        }                                                                                                                
}

... Everything was working fine, so I wanted more! I started appending our stream metadata to get what was listening to at a given time. It could be an ad, a radio show, a track from our playlist, a jingle, etc. The input file started to look like this:

...
138.255.255.224#WinampMPEG/5.50#72596#5164253#ADS
144.255.255.70#VLC/2.1.4 LibVLC/2.1.4#14647#5215414#LINER
78.255.255.203#fbxmms/1.0 FreeboxPlayer/6.0 (like AppleWebKit)#5591#5218475#BOBBY'S-AWESOME_RADIO SHOW
78.196.255.255#Lavf53.32.100#2857#5219883#HOURLY JINGLE
92.255.255.93#VLC/2.2.6 LibVLC/2.2.6#2653#5220001#ARTIST WITH SPACES#SONGTITLE WITH SPACES
...

So, I set up my grok filter accordingly, or at least I tried to:

filter
{
grok {
match => { "message" => [
"%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{DATA:Artist}#%{DATA:SongTitle}",
"%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{DATA:BAJingleAdOrShow}",
"%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}"
]
}
}
...
}

Here comes my issue: from that filter configuration, I see in Kibana that I got a new indexed field: Artist. But so far, it fails to create/add the SongTitle one. I tried switching DATA to SPACE, but it didn't do anything.

Our radio schedule won't air any radio show until this evening, so I figured that's why the BAJingleAdOrShow doesn't show up yet in Kabana field's list, so I do not worry with that yet.

Why would the file parsing filter partially work? How could it create/update (in elasticsearch) the Artist field without creating/updating the SongTitle field at the same time?

Best Regards from France,

2

Good evening,

As it turned out, the second possibility "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{DATA:BAJingleAdOrShow} isn't parsed, and it falls back to "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}" ; I still doesn't retrieve the SongTitle as a indexed field.

So to sum it up :

92.255.255.93#VLC/2.2.6 LibVLC/2.2.6#2653#5220001#ARTIST WITH SPACES#SONGTITLE WITH SPACES

this is parsed with "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{DATA:Artist}#%{DATA:SongTitle}" but the SongTitle isn't shown within Kibana (but Artist is)

138.255.255.224#WinampMPEG/5.50#72596#5164253#ADS
144.255.255.70#VLC/2.1.4 LibVLC/2.1.4#14647#5215414#LINER
78.255.255.203#fbxmms/1.0 FreeboxPlayer/6.0 (like AppleWebKit)#5591#5218475#BOBBY'S-AWESOME_RADIO SHOW
78.196.255.255#Lavf53.32.100#2857#5219883#HOURLY JINGLE

those lines aren't parsed with "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{DATA:BAJingleAdOrShow} since the field BAJingleAdOrShow isn't shown within Kibana (even after refreshing the index in Management) but the data are still output within elasticsearch with the "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}" pattern.

I can't seem to figure what is going wrong here ; do you have any clue?

Best Regards

3

Hello there,

So I managed to get a config that works, so I am posting it right there before updating this thread as saolved :slight_smile:

Thank you for your concern!

filter
{
grok {
match => [
"message", "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{DATA:Artist}#%{USER:SongTitle}",
"message", "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}#%{USER:BAJingleAdOrShow}",
"message", "%{IPORHOST:IP}#%{DATA:UserAgent}#%{NUMBER:TimeConnected}#%{NUMBER:ID}"
]
}
}

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.