Dissector mapping field not found in event

Elastic Stack Logstash
1

Hi, I used the dissect filter and it worked .Now it does not.
The only change is the upgrate of mac Os Catalina to mac OS Big Sur v 11.4
A message is displayed : Dissector mapping field not found in event
Could you help me.

PG

2

Can you post your config/sample data and a few log messages from logstash?

Dissector mapping field not found in event means that the field that you are trying to dissect is not in the data that is being ingested.

Have you looked at your data in the standard output to ensure your incoming data has not changed since your upgrade?

3

here my logstash config :input{
stdin{}
}

filter {
dissect {
mapping => {
"message" => '%{?messageBis} <form %{messageBis1}'
"messageBis1" => '%{field1}%{?field2}%{field3}%{?field4}%{field5}'

}

remove_field => ["message"]

}

}

output {

stdout { codec => rubydebug }

}

4

And a part of my sample

Créée le  2013-05-23 09:32:20
Libellé  thermocouples
Etat actuel  0
Commentaire
Service  /DR/ICAA
Nom du rédacteur  Sawsane NAKOUZI
Numéro commande  
Bénéficiaire
Sawsane NAKOUZI
Délai de livraison  91468
Code analytique principal  ARMINES-CROMEP
Code analytique secondaire  
Pieces jointes Aucune
Informations sur le fournisseur 
Nom  NEWPORT OMEGA
Adresse * 11 rue Jacques Cartier 78280 GUYANCOURT France
Téléphone  0161372900
Fax  0130575427
Type de la DA
ARMINES
CNRS
Ctre commun Armines  
No Contrat  92751
Nom Contrat  Toshiba
Financement (Inv.DR)  
Achat relevant de la procédure d'hygiène et sécurité OUI NON
N° Désignation Code nomenclature Référence Quantité PU HT TVA Total HT Total TTC
1 Alumel
 
TFAL-003-100 1.00 29.00 19.6 29.00 34.68
2 chromel
 
TFCY-003-100 1.00 29.00 19.6 29.00 34.68
TOTAUX (EUR)
11.36
58
69.36
ATTENTION, Cette application n'est a utiliser QUE POUR LES DA CNRS ou ARMINES RAPSODEE, mais PAS POUR CELLES ARMINES ICAA OU EPA !!!
Pour TOUTES les autres DA : https://sirepa-prod.mines-telecom.fr
HistoriqueCopier
Informations sur la procédure de consultation (pour DA de montant > 4000 EUR HT)
Descriptif technique
Fournisseurs consultés
Raisons du choix du fournisseur
Historique de la DA No 17758 à partir de l'état 1
Date Auteur Commentaire Etat final

5

It worked, why it does not ?

6

My guess is that this field is the one that is complaining about because messageBis1 is not a standard field. It may be a good idea to disable all filters and just have an input/output in your logstash config and see what logstash is getting.

7

I get the differents fields indicated in the config and I have to keep on going the sample 'dissection.

{
"@version" => "1",
"field3" => "\n<td valign=\"top\">\n<table class=\"DA_DG\">\n<th class=\"soustitre\" colspan=2>Informations sur le fournisseur \n<input class=\"petit_bouton\" type=button onClick=\"recherche('fournisseur')\" value=\"...\">\n<td class=\"libelle\">Nom\n <input type=hidden id=\"four_nom\" name=\"four_nom\" value=\"GROSSERON SAS\">\n \n <td class=\"lecture_seule\"> GROSSERON SAS\n <td class=\"libelle\">Adresse *\n<textarea class='normal' name='four_adr' id='four_adr'rows=3\">ZAC Hauts de Coueron III - Secteur 4\r\n4 rue des entrepreneurs\r\n44220 COU\xCBRON - FRANCE<td class=\"libelle\">T\xE9l\xE9phone\n <input type=hidden id=\"four_tel\" name=\"four_tel\" value=\"33 (0) 2 40 92 07 09\">\n \n <td class=\"lecture_seule\"> 33 (0) 2 40 92 07 09\n <td class=\"libelle\">Fax\n <input type=hidden id=\"four_fax\" name=\"four_fax\" value=\"33 (0) 2 40 92 07 10\">\n \n <td class=\"lecture_seule\"> 33 (0) 2 40 92 07 10\n ",
"@timestamp" => 2021-03-19T16:45:01.499Z,
"field5" => "<td class=\"libelle\">Ctre commun Armines\n <input type=hidden id=\"ccarmines\" name=\"ccarmines\" value=\"\">\n \n <td class=\"lecture_seule\"> \n <td class=\"libelle\">No Contrat\n <input type=hidden id=\"code_contrat\" name=\"code_contrat\" value=\"20000270\">\n \n <td class=\"lecture_seule\"> 20000270\n <td class=\"libelle\">Nom Contrat\n <input type=hidden id=\"nom_contrat\" name=\"nom_contrat\" value=\"MAANEO\">\n \n <td class=\"lecture_seule\"> MAANEO\n <td class=\"libelle\">Financement (Inv.DR)\n <input type=hidden id=\"financement_dr\" name=\"financement_dr\" value=\"\">\n \n <td class=\"lecture_seule\"> \n <td colspan=\"2\"><table width=\"100%\" bgcolor=\"pink\">\n\n\n\n

Achat relevant de la proc\xE9dure d'hygi\xE8ne et s\xE9curit\xE9 <input name='hygiene' type=radio value=\"1\">OUI <input name='hygiene' type=radio value=\"0\" CHECKED>NON
\n\n<table width=\"100%\" border=\"1\">\n<tr class=\"soustitre\">\nN\xB0\nD\xE9signation\nCode nomenclature\nR\xE9f\xE9rence\nQuantit\xE9\nPU HT\nTVA\nTotal HT\nTotal TTC\n\n<tr id=\"ligne_article_1\" class=\"article\"><input type='hidden' name='id_article_0' value=\"51526\">1BAC DE RETENTION LABO-RETEN 60L AVEC CAILLEBOTIS PLASTIQUE GRIS ARGENT<table border=\"1\">\n <input type=hidden id=\"nomenclature_code_0\" name=\"nomenclature_code_0\" value=\"\">\n <td class=\"lecture_seule\"> \n B032927\n 1.00\n <td width=\"2em\">121.16\n 20\n 121.16\n 145.39\n \n <tr id=\"ligne_article_2\" class=\"article\"><input type='hidden' name='id_article_1' value=\"51527\">2Frais de transport<table border=\"1\">\n <input type=hidden id=\"nomenclature_code_1\" name=\"nomenclature_code_1\" value=\"\">\n <td class=\"lecture_seule\"> \n 708501\n 1.00\n <td width=\"2em\">22.00\n 20\n 22.00\n 26.40\n \n \n <th colspan=\"2\">\n <input type=\"button\" class=\"bouton\" onClick=\"montrer();\" id=\"20lignes\" value=\"20 lignes\">\n<input type=\"button\" class=\"petit_bouton\" onClick=\"cacher();\" id=\"10lignes\" value=\"10 lignes\">\n<th colspan=\"4\">\nTOTAUX (EUR)\n<th width=\"2em\" align=\"right\" bgcolor=\"orange\">
28.63
\n<th width=\"2em\" align=\"right\" bgcolor=\"orange\">
143.16
\n<th width=\"2em\" align=\"right\" bgcolor=\"orange\">
171.79
\n<input type=hidden name=\"totalHT\" id=\"totalHT\" value='0'>\n<input type=hidden name=\"totalTVA\" id=\"totalTVA\" value='0'>\n<input type=hidden name=\"totalTTC\" id=\"totalTTC\" value='0'>\n<th bgcolor=\"red\" colspan=\"10\">ATTENTION, Cette application n'est a utiliser QUE POUR LES DA CNRS ou ARMINES RAPSODEE, mais PAS POUR CELLES ARMINES ICAA OU EPA !!!\n <th bgcolor=\"red\" colspan=\"10\">Pour TOUTES les autres DA : \n <a href=\"https://sirepa-prod.mines-telecom.fr\">https://sirepa-prod.mines-telecom.fr<td align=\"center\" colspan=\"10\"><a href=\"da.cgi?IdDa=24530&Action=HistoriqueDemande\" class=\"bouton\">Historique<a href=\"da.cgi?IdDa=24530&Action=CopierDemande\" class=\"bouton\">Copier\n <th class=\"soustitre\" colspan=2>Informations sur la proc\xE9dure de consultation (pour DA de montant > 4000 EUR HT)\n<tr valign=\"top\">\n<table class=\"DA_DG\">\n<td class=\"libelle\">Descriptif technique\n<textarea class=\"saisie\" name=\"descriptif\" id=\"descriptif\" rows=9>\n\n<table class=\"DA_DG\">\n<td class=\"libelle\">Fournisseurs consult\xE9s\n<textarea class=\"saisie\" name=\"fournisseurs\" id=\"fournisseurs\" rows=3>\n<td class=\"libelle\">Raisons du choix du fournisseur\n<textarea class=\"saisie\" name=\"raisons\" id=\"raisons\" rows=5>\n\n<td colspan=\"2\"><table class=\"historique\">\n <th class=\"titre\" colspan=\"4\">\n Historique de la DA No 24530 \xE0 partir de l'\xE9tat 1\n \n DateAuteurCommentaireEtat final\n Le 03/02/2021 \xE0 09:48\n \n Changement \xE9tat\n 1 (en attente de validation par responsable de niveau 1)\n \n \n Le 03/02/2021 \xE0 10:38\n \n mise_a_jour\n 1 (en attente de validation par responsable de niveau 1)\n \n
<input class=\"bouton\" type=button onClick=\"javascript: history.back();\" value=\"Retour\">\n\n<script langage=\"javascript\">calculerPrix(this.form);\n \n ",

etc ...

8

I got and now it does not work.

9

Oh I see.
Can you please post more detailed log entries of the error you are getting?

10

Sure.
Here more details :slight_smile:

WARN ][org.logstash.dissect.Dissector][main] Dissector mapping, field found in event but it was empty {"field"=>"message", "event"=>{"host"=>"I5951.local", "message"=>"", "@version"=>"1", "@timestamp"=>2021-07-20T13:33:15.563Z, "tags"=>["_dissectfailure"]}}
[2021-07-20T15:33:16,933][WARN ][org.logstash.dissect.Dissector][main] Dissector mapping, field not found in event {"field"=>"messageBis1", "event"=>{"host"=>"I5951.local", "message"=>"", "@version"=>"1", "@timestamp"=>2021-07-20T13:33:15.563Z, "tags"=>["_dissectfailure"]}}
[2021-07-20T15:33:16,943][WARN ][org.logstash.dissect.Dissector][main] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{?messageBis} <form %{messageBis1}", "event"=>{"host"=>"I5951.local", "message"=>" ", "@version"=>"1", "@timestamp"=>2021-07-20T13:33:15.609Z, "tags"=>["_dissectfailure"]}}
[2021-07-20T15:33:16,946][WARN ][org.logstash.dissect.Dissector][main] Dissector mapping, field not found in event {"field"=>"messageBis1", "event"=>{"host"=>"I5951.local", "message"=>" ", "@version"=>"1", "@timestamp"=>2021-07-20T13:33:15.609Z, "tags"=>["_dissectfailure"]}}
[2021-07-20T15:33:16,950][WARN ][org.logstash.dissect.Dissector][main] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{?messageBis} <form %{messageBis1}", "event"=>{"host"=>"I5951.local", "message"=>" ", "@version"=>"1", "@timestamp"=>2021-07-20T13:33:15.609Z, "tags"=>["_dissectfailure"]}}

11

The first indicates, as it says, that the [message] is empty.

The second suggests that you are assuming that your dissect always works, and then trying a second dissect even when the field it tries to dissect was not created by the first dissect. You should check whether the field exists.

The third tell you that the [message] field contains a single space, so the pattern does not match.

12

Yes the pattern does not work now but it did with the same sample and the same code.The only change I do is the upgrade of mac OS.coud it be the reason of that?

13

OK I will come back to work

14

Night advises.Thank tou very much for your answers.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.