[Logstash] CSP-REPORT - URI non parsé

Bonjour !

Le contexte :
J'importe actuellement un log de rapport CSP dans Elasticseach. Je cherche à splitter le domaine de l'url (URI)

Voici un exemple de JSON importé :

{"csp-report":{
"blocked-uri":"self",
"document-uri":"https://env.application.fr/toto",
"line-number":762,
"original-policy":"default-src 'none'; connect-src https://env.application.fr:443; font-src https://env.application.fr:443; img-src https://env.application.fr:443; script-src https://env.application.fr:443 'unsafe-inline'; style-src https://env.application.fr:443 'unsafe-inline'; report-uri https://appli/csp-cisirh.php",
"referrer":"https://env.application.fr/toto",
"script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.application.fr/js/a2d8d78.js",
"violated-directive":"script-src https://env.application.fr:443 'unsafe-inline'"
}
}

L'import se passe bien, j'extrais ensuite la valeur document-uri dans une variable uri (mutate add_fied uri):

filter {
 ### STATISTIQUES RAPPORT CSP
     if  "CSP" in [tags]  {
        ### Parsing JSON
        json{ source => "message"}
        mutate { add_field => { "uri" => "%{[csp-report][document-uri]}" } }
        grok{
            patterns_dir => ["/produit/logstash/patterns"]
            match => { "uri" => "%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?"}
        }
     }
}

J'essaie ensuite de parser ce champs uri avec les motifs %{URI} ou %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})? , un motif personnalisé etc..

Et je n'obtiens que des _grokparsefailure, pourtant les motifs URI* sont bien chargées au démarrage de logstash..

[2017-10-31T17:46:16,328][DEBUG][logstash.pipeline        ] 
output received {"event"=>{"offset"=>2904, 
"source"=>"/exploit/httpd-piwik/logs/csp-reports.log", "program"=>"httpd", "message"=>"{\"csp-report\":{\"blocked-uri\":\"self\",\"document-uri\":\"https://env.application.fr/agent/927/import-crep-papier\",
\"line-number\":762,\"original-policy\":\"default-src none; connect-src https://env.application.fr:443; font-src https://env.application.fr:443; img-src https://env.application.fr:443; script-src https://env.application.fr:443 unsafe-inline; style-src https://env.application.fr:443 unsafe-inline; report-uri https://serveur-csp.frcsp-cisirh.php\",
\"referrer\":\"https://env.application.fr/agent/927/import-crep-papier\",
\"script-sample\":\"call to eval() or related function blocked by CSP\",
\"source-file\":\"https://env.application.fr/js/a2d8d78.js\",
\"violated-directive\":\"script-src https://env.application.fr:443 unsafe-inline\"}}", 
"type"=>"applicatif", 
"csp-report"=>{"referrer"=>"https://env.application.fr/agent/927/import-crep-papier",
 "script-sample"=>"call to eval() or related function blocked by CSP",
 "original-policy"=>"default-src none; connect-src https://env.application.fr:443; font-src https://env.application.fr:443; img-src https://env.application.fr:443; script-src https://env.application.fr:443 unsafe-inline; style-src https://env.application.fr:443 unsafe-inline; report-uri https://serveur-csp.fr.php",
 "source-file"=>"https://env.application.fr/js/a2d8d78.js",
 "violated-directive"=>"script-src https://env.application.fr:443 unsafe-inline", 
"document-uri"=>"https://env.application.fr/agent/927/import-crep-papier", 
"line-number"=>762, 
"blocked-uri"=>"self"}, 
uri"=>"https://env.application.fr/agent/927/import-crep-papier", 
"tags"=>["json", "CSP", "_grokparsefailure"], 
"@timestamp"=>2017-10-31T16:46:15.063Z, "@version"=>"1", 
"beat"=>{"name"=>"hostname", "hostname"=>"hostname", "version"=>"5.6.1"},
"host"=>"hostname", 
"fields"=>{"type"=>"CSP", "environment"=>"PRODJ"}}}

Merci pour votre aidre, j'y perds mon GROK ! :sweat_smile:

David

EDIT : ajout de retour-chariots pour la lecture du JSON..

Idéalement, ça serait bien de faire un exemple beaucoup plus simple avec un document JSON contenant le texte que Grok doit lire et juste une config logstash comme:

input { stdin {} }

filter {
  grok {
    ...
  }
}

output { stdout { codec => rubydebug } }

puis

echo "YOUR CONTENT" | logstash -f logstash.conf

Ca permettrait de supprimer tout le bruit dans ton exemple.

Ensuite je te conseille de simplement utiliser quelque chose comme:

Bonjour et merci pour ta réponse !

J'utilise Grok Debugger pour tester mes motifs, c'est un super outil. J'ai essayé d'utiliser l'outil présent dans Kibana mais j'ai tendance à bien faire planter Kibana avec.

Donc je résume :
1 - j'ai 1 fichier à parser avec X lignes formatées en JSON
2 - Le fichier est parcouru par filebeat
3 - Logstash reçoit la ligne JSON
3.1 - Split de la ligne json
json{ source => "message"}
3.2 récupération de la valeur document-uri
mutate { add_field => { "uri" => "%{[csp-report][document-uri]}" } }
3.3 Parsing du champ uri
grok { match => { "uri" => "%{URI}"} }
4 . envoi vers elasticsearch

Les étapes 1,2 , 3.1 , 3.2 et 4 fonctionnent parfaitement sans l'étape 3.3.

Le souci est au niveau du 3.3 (et impacte le 4 ensuite).

Fichier 00-filter-json.conf

input { stdin {} }

filter {
  json{ source => "message"}
  mutate { add_field => { "uri" => "%{[csp-report][document-uri]}" } }
  grok { match => { "uri" => "%{URI}"} }
}

output { stdout { codec => rubydebug } }

Voici la sortie (un peu formatée) :

Chargement des filtres

[2017-11-02T10:37:24,911][DEBUG][logstash.filters.grok    ] Adding pattern {"URIPROTO"=>"[A-Za-z]([A-Za-z0-9+\\-.]+)+"}
[2017-11-02T10:37:24,911][DEBUG][logstash.filters.grok    ] Adding pattern {"URIHOST"=>"%{IPORHOST}(?::%{POSINT:port})?"}
[2017-11-02T10:37:24,911][DEBUG][logstash.filters.grok    ] Adding pattern {"URIPATH"=>"(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%&_\\-]*)+"}
[2017-11-02T10:37:24,912][DEBUG][logstash.filters.grok    ] Adding pattern {"URIPARAM"=>"\\?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\\-\\[\\]<>]*"}
[2017-11-02T10:37:24,912][DEBUG][logstash.filters.grok    ] Adding pattern {"URIPATHPARAM"=>"%{URIPATH}(?:%{URIPARAM})?"}
[2017-11-02T10:37:24,912][DEBUG][logstash.filters.grok    ] Adding pattern {"URI"=>"%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?"}

[2017-11-02T10:37:25,124][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Entrée de l'event (3.1):

[2017-11-02T10:37:25,176][DEBUG][logstash.filters.json    ] Event after json filter {:event=>2017-11-02T09:37:25.092Z hostname {"csp-report":{"blocked-uri":"self","document-uri":"https://env.appli.orga/agent/927/import-crep-papier","line-number":762,"original-policy":"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php","referrer":"https://env.appli.orga/agent/927/import-crep-papier","script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.appli.orga/js/a2d8d78.js","violated-directive":"script-src https://env.appli.orga:443 unsafe-inline"}}}

Mutate add field uri (3.2):

[2017-11-02T10:37:25,179][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"uri", "value"=>["%{[csp-report][document-uri]}"]}

Grok (3.3)

[2017-11-02T10:37:25,180][DEBUG][logstash.filters.grok    ] Running grok filter {:event=>2017-11-02T09:37:25.092Z hostname {"csp-report":{"blocked-uri":"self","document-uri":"https://env.appli.orga/agent/927/import-crep-papier","line-number":762,"original-policy":"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php","referrer":"https://env.appli.orga/agent/927/import-crep-papier","script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.appli.orga/js/a2d8d78.js","violated-directive":"script-src https://env.appli.orga:443 unsafe-inline"}}}

Sortie (4)

[2017-11-02T10:37:25,182][DEBUG][logstash.filters.grok    ] Event now:  {:event=>2017-11-02T09:37:25.092Z hostname {"csp-report":{"blocked-uri":"self","document-uri":"https://env.appli.orga/agent/927/import-crep-papier","line-number":762,"original-policy":"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php","referrer":"https://env.appli.orga/agent/927/import-crep-papier","script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.appli.orga/js/a2d8d78.js","violated-directive":"script-src https://env.appli.orga:443 unsafe-inline"}}}
[2017-11-02T10:37:25,183][DEBUG][logstash.pipeline        ] output received {"event"=>{"@timestamp"=>2017-11-02T09:37:25.092Z, "@version"=>"1", "host"=>"hostname", "message"=>"{\"csp-report\":{\"blocked-uri\":\"self\",\"document-uri\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"line-number\":762,\"original-policy\":\"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php\",\"referrer\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"script-sample\":\"call to eval() or related function blocked by CSP\",\"source-file\":\"https://env.appli.orga/js/a2d8d78.js\",\"violated-directive\":\"script-src https://env.appli.orga:443 unsafe-inline\"}}", "csp-report"=>{"referrer"=>"https://env.appli.orga/agent/927/import-crep-papier", "script-sample"=>"call to eval() or related function blocked by CSP", "original-policy"=>"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php", "source-file"=>"https://env.appli.orga/js/a2d8d78.js", "violated-directive"=>"script-src https://env.appli.orga:443 unsafe-inline", "document-uri"=>"https://env.appli.orga/agent/927/import-crep-papier", "line-number"=>762, "blocked-uri"=>"self"}, "uri"=>"https://env.appli.orga/agent/927/import-crep-papier"}}

Output :

{
    "@timestamp" => 2017-11-02T09:37:25.092Z,
      "@version" => "1",
          "host" => "hostname",
       "message" => "{\"csp-report\":{\"blocked-uri\":\"self\",\"document-uri\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"line-number\":762,\"original-policy\":\"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php\",\"referrer\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"script-sample\":\"call to eval() or related function blocked by CSP\",\"source-file\":\"https://env.appli.orga/js/a2d8d78.js\",\"violated-directive\":\"script-src https://env.appli.orga:443 unsafe-inline\"}}",
    "csp-report" => {
                  "referrer" => "https://env.appli.orga/agent/927/import-crep-papier",
             "script-sample" => "call to eval() or related function blocked by CSP",
           "original-policy" => "default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php",
               "source-file" => "https://env.appli.orga/js/a2d8d78.js",
        "violated-directive" => "script-src https://env.appli.orga:443 unsafe-inline",
              "document-uri" => "https://env.appli.orga/agent/927/import-crep-papier",
               "line-number" => 762,
               "blocked-uri" => "self"
    },
           "uri" => "https://env.appli.orga/agent/927/import-crep-papier"

Arrêt de logstash

[2017-11-02T10:37:25,198][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x381d30ad>
[2017-11-02T10:37:25,198][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x6cb89d08>
[2017-11-02T10:37:25,198][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x6a30349d>
[2017-11-02T10:37:25,198][DEBUG][logstash.filters.json    ] closing {:plugin=>"LogStash::Filters::Json"}
[2017-11-02T10:37:25,199][DEBUG][logstash.filters.mutate  ] closing {:plugin=>"LogStash::Filters::Mutate"}
[2017-11-02T10:37:25,199][DEBUG][logstash.filters.grok    ] closing {:plugin=>"LogStash::Filters::Grok"}
[2017-11-02T10:37:25,385][DEBUG][logstash.outputs.stdout  ] closing {:plugin=>"LogStash::Outputs::Stdout"}
[2017-11-02T10:37:25,385][DEBUG][logstash.pipeline        ] Pipeline main has been shutdown
[2017-11-02T10:37:28,027][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-11-02T10:37:28,028][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-11-02T10:37:28,028][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-11-02T10:37:28,033][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-11-02T10:37:28,034][DEBUG][logstash.pipeline        ] Closing inputs
[2017-11-02T10:37:28,034][DEBUG][logstash.inputs.stdin    ] stopping {:plugin=>"LogStash::Inputs::Stdin"}
[2017-11-02T10:37:28,034][DEBUG][logstash.pipeline        ] Closed inputs

En fait, on dirait que GROK ne passe pas du tout sur le champ "uri" préalablement défini mais reste sur l'event global.. ?!
Au fur et à mesure de mes essais, j'ai pu voir que grok parsait le champ "message" en lieu et place du champ passé en paramètre : :face_with_raised_eyebrow:

Test de la configuration :slight_smile:

[2017-11-02T13:46:49,445][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"grok", :type=>"filter", :class=>LogStash::Filters::Grok}
[2017-11-02T13:46:49,451][DEBUG][logstash.filters.grok    ] config LogStash::Filters::Grok/@match = {"uri"=>"%{URI}"}
[2017-11-02T13:47:11,863][DEBUG][logstash.filters.mutate  ] filters/LogStash::Filters::Mutate: removing field {:field=>"message"}
[2017-11-02T13:47:11,864][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"uri", "value"=>["%{[csp-report][document-uri]}"]}
[2017-11-02T13:47:11,865][DEBUG][logstash.filters.grok    ] Running grok filter {:event=>2017-11-02T12:47:11.738Z hostname %{message}}
[2017-11-02T13:47:11,868][DEBUG][logstash.filters.grok    ] Event now:  {:event=>2017-11-02T12:47:11.738Z hostname %{message}}

J'ai donc cherché à remplacer le champ message par le contenu à parser :

input { stdin {} }

filter {
  json { source => "message"}
  mutate {remove_field => "message"}
  mutate { add_field => { "message" => "%{[csp-report][document-uri]}" } }
  grok { match => { "message" => "%{URI}"} }
}

output { stdout { codec => rubydebug } }

le résultat n'a pas été concluant. :sweat:

J'ai aussi essayé de d'accèder directement aux données du JSON :
grok { match => { "%{[csp-report][document-uri]" => "%{URI}" } }
ou
grok { match => { "%{[message][csp-report][document-uri]" => "%{URI}" } }

J'ai vraiment l'impression que le seul champ que Grok peut parser est le champ "message"

Je pense que je vais finir par utiliser insérer du code ruby pour m'en sortir..

David

Peut-être que @colinsurprenant a une idée?

Je n'ai pas précisé mais je suis en 5.6.1

J'ai fini par utiliser un bout de code ruby qui marche plutôt bien. :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.