Filter grok : Few lines differents in log file

Elastic Stack Logstash
1

Hello,

I have a little problem about performing of of my "q_compt" file (file text)

80% of lines have 4 fields
10% have 3 fields
10% have 4 fields but have differents informations in.

Here my filter expression :

if [type] == "q_compt" {
     grok {
         match => { "message" => "%{DATA:date}[;]%{DATA:name_compt}[;]%{DATA:applications}[;]%{NUMBER:volume}" }
          }

Here an example of lines :

201702;Vol-EDR-2;appli1;15000 (4 fields)
201702;Vol-EDR-2;appli2;5000 (4 fields)
201702;Vol-EDR-total;20000 (3 fields)

201702;CA-EDR-2;appli1;850 (4 fields)
201702;CA-EDR-2;appli2;150 (4 fields)
201702;CA-EDR-total;1000 (3 fields)

I had thought this :

if [type] == "q_compt" {
     grok {
         match => { "message" => "%{DATA:date}[;]%{DATA:name_compt}[;]%{DATA:applications}[;]%{NUMBER:volume}"    ||  "%{DATA:date}[;]%{DATA:name_compt}[;]%{NUMBER:volume_total}" }
 }
2

I have about 30 differents name_compt (field n°2) in the one log file text

I had thought cut the main file into as many files as there are name_compt.

So i should modified the filebeat prospect as :

filebeat.prospectors:
  - input_type: log
    - "/log_file_applis/Vol-EDR-2"
         document_type: Vol-EDR-2

    - "/log_file_applis/CA-EDR-2"
         document_type: CA-EDR-2

So, in logstash conf :

if [type] == "Vol-EDR-2" {
     grok {...}}

if [type] == "CA-EDR-2" {
    grok {...}}

But this way of doing involves do write 30 type of prospector and 30 "if [type] in logstash conf.

3

Why not just have a single grok filter with two expressions, one that matches the four-field case and one that matches the three-field case? Logstash will match them in turn, stopping at the first match.

4

I'm stupid, this is a simply way !

Do you know how change octet values to gigaoctet values ? It's in Kibana or directly in logstash conf ?

Thank you for your answer

5

Do you know how change octet values to gigaoctet values ? It's in Kibana or directly in logstash conf ?

Not sure if you can do it in Kibana but it's definitely possible with Logstash. The units filter seems to be able to do it but otherwise a ruby filter can be used.