Create and split some fields

Elastic Stack Logstash
1

Hello,

I have some lines in an EAD.stats.log file like that :

DocType CNI: # DateEmission = 100% # DateNaissance = 100% # DateValidite = 100% # DepartementDelivrance = 100% # LieuNaissance = 10% # MRZ = 100% # MRZ2 = 100% # Nom = 100% # Numero = 100% # Prenom = 100% # Sexe = 100%

I would like to create many fields with this line (field 1 : DateEmission, field 2 : 100%, field 3 : DateNaissance, field 4 : 100%...) but the number of fields is variable from one line to another.

Is it possible to do that? Or maybe could I take the whole line and split it after?

I'm working with Logstash 2.1.1.

Thanks !

2

Apart from DocType CNI: # this looks like a key-value list. You could use grok to capture everything after the initial sequence in a separate field and then apply the kv filter to this in order to parse the data.

3

Hello,

Thank you for your reply. I tested with this code but it did not work.

grok{
match => [ "message", "%{DATA}|[%{DATA:processId}]|[%{DATA:level}]|[%{DATA}]|%{DATA}|(%{DATA:method}) |%{DATA:appClientId}|%{DATA:coetb}|%{DATA:clientId}|%{DATA}:%{DATA}d %{DATA:DocId} - %{DATA} p - DocType %{DATA:DocType}: # %{DATA:INDEXString}|%{GREEDYDATA:message}" ]
overwrite => [ "message" , "level", "processId", "method", "appClientId", "coetb", "clientId", "DocId", "DocType", "INDEXString"]
}
if [field] == "INDEXString" {
kv {
field_split => "#?"
}
}

The separation didn't work :

INDEXString DateEmission = 100% # DateNaissance = 100% # DateValidite = 100% # DepartementDelivrance = 100% # LieuNaissance = 10% # MRZ = 100% # MRZ2 = 100% # Nom = 100% # Numero = 100% # Prenom = 100% # Sexe = 100%

4

That is not what I tried to describe. Assuming that the initial part of the log is static, I was thinking about something like this:

grok {
  match => { "message" => "DocType CNI: # %{GREEDYDATA:kv_list}" }
}

kv {
  source => "kv_list"
  field_split => " # "
  value_split => " = "
  remove_field => ["kv_list"]
}
5

Thank you, it works. I had forgotten to add the source. Have a good day!

6

image.png575×599 22.4 KB

I still need your help please. I would like to create a new event for each new field I split.

For example :

One event with timestamp, version, docId, DocType, ligne, NombrePieces (there are mandatory fields) and DateEmission (the first split field).

One event with timestamp, version, docId, DocType, ligne, NombrePieces and DateNaissance (the second split field).

One event with timestamp, version, docId, DocType ligne, NombrePieces and DateValidite (the third split field).

...

7

Why do you want to create separate documents like that?

8

Because I need to agreegate each events in Kibana differently.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.