Create and split some fields


(Jean Marc Glevarec) #1

Hello,

I have some lines in an EAD.stats.log file like that :

DocType CNI: # DateEmission = 100% # DateNaissance = 100% # DateValidite = 100% # DepartementDelivrance = 100% # LieuNaissance = 10% # MRZ = 100% # MRZ2 = 100% # Nom = 100% # Numero = 100% # Prenom = 100% # Sexe = 100%

I would like to create many fields with this line (field 1 : DateEmission, field 2 : 100%, field 3 : DateNaissance, field 4 : 100%...) but the number of fields is variable from one line to another.

Is it possible to do that? Or maybe could I take the whole line and split it after?

I'm working with Logstash 2.1.1.

Thanks !


(Christian Dahlqvist) #2

Apart from DocType CNI: # this looks like a key-value list. You could use grok to capture everything after the initial sequence in a separate field and then apply the kv filter to this in order to parse the data.


(Jean Marc Glevarec) #3

Hello,

Thank you for your reply. I tested with this code but it did not work.

grok{
match => [ "message", "%{DATA}|[%{DATA:processId}]|[%{DATA:level}]|[%{DATA}]|%{DATA}|(%{DATA:method}) |%{DATA:appClientId}|%{DATA:coetb}|%{DATA:clientId}|%{DATA}:%{DATA}d %{DATA:DocId} - %{DATA} p - DocType %{DATA:DocType}: # %{DATA:INDEXString}|%{GREEDYDATA:message}" ]
overwrite => [ "message" , "level", "processId", "method", "appClientId", "coetb", "clientId", "DocId", "DocType", "INDEXString"]
}
if [field] == "INDEXString" {
kv {
field_split => "#?"
}
}

The separation didn't work :

INDEXString DateEmission = 100% # DateNaissance = 100% # DateValidite = 100% # DepartementDelivrance = 100% # LieuNaissance = 10% # MRZ = 100% # MRZ2 = 100% # Nom = 100% # Numero = 100% # Prenom = 100% # Sexe = 100%


(Christian Dahlqvist) #4

That is not what I tried to describe. Assuming that the initial part of the log is static, I was thinking about something like this:

grok {
  match => { "message" => "DocType CNI: # %{GREEDYDATA:kv_list}" }
}

kv {
  source => "kv_list"
  field_split => " # "
  value_split => " = "
  remove_field => ["kv_list"]
}

(Jean Marc Glevarec) #5

Thank you, it works. I had forgotten to add the source. Have a good day!


(Jean Marc Glevarec) #6

I still need your help please. I would like to create a new event for each new field I split.

For example :

One event with timestamp, version, docId, DocType, ligne, NombrePieces (there are mandatory fields) and DateEmission (the first split field).

One event with timestamp, version, docId, DocType, ligne, NombrePieces and DateNaissance (the second split field).

One event with timestamp, version, docId, DocType ligne, NombrePieces and DateValidite (the third split field).

...


(Christian Dahlqvist) #7

Why do you want to create separate documents like that?


(Jean Marc Glevarec) #8

Because I need to agreegate each events in Kibana differently.


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.