# \[Logstash\] CSP-REPORT - URI non parsé
**URL:** https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930
**Category:** Discussions en français
**Created:** [October 31, 2017, 4:52pm UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930 "2017-10-31T16:52:35Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@David\_ROELANDT](https://discuss.elastic.co/u/David_ROELANDT)
#### Post date: [October 31, 2017, 4:52pm UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/1 "2017-10-31T16:52:35Z")
Bonjour !
Le contexte :
J'importe actuellement un log de rapport CSP dans Elasticseach. Je cherche à splitter le domaine de l'url (URI)
Voici un exemple de JSON importé :
```json
{"csp-report":{
"blocked-uri":"self",
"document-uri":"https://env.application.fr/toto",
"line-number":762,
"original-policy":"default-src 'none'; connect-src https://env.application.fr:443; font-src https://env.application.fr:443; img-src https://env.application.fr:443; script-src https://env.application.fr:443 'unsafe-inline'; style-src https://env.application.fr:443 'unsafe-inline'; report-uri https://appli/csp-cisirh.php",
"referrer":"https://env.application.fr/toto",
"script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.application.fr/js/a2d8d78.js",
"violated-directive":"script-src https://env.application.fr:443 'unsafe-inline'"
}
}
```
L'import se passe bien, j'extrais ensuite la valeur document-uri dans une variable uri (mutate add\_fied uri):
```ruby
filter {
### STATISTIQUES RAPPORT CSP
if "CSP" in [tags] {
### Parsing JSON
json{ source => "message"}
mutate { add_field => { "uri" => "%{[csp-report][document-uri]}" } }
grok{
patterns_dir => ["/produit/logstash/patterns"]
match => { "uri" => "%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?"}
}
}
}
```
J'essaie ensuite de parser ce champs uri avec les motifs `%{URI}` ou `%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?` , un motif personnalisé etc..
Et je n'obtiens que des \_grokparsefailure, pourtant les motifs URI\* sont bien chargées au démarrage de logstash..
```json
[2017-10-31T17:46:16,328][DEBUG][logstash.pipeline]
output received {"event"=>{"offset"=>2904,
"source"=>"/exploit/httpd-piwik/logs/csp-reports.log", "program"=>"httpd", "message"=>"{\"csp-report\":{\"blocked-uri\":\"self\",\"document-uri\":\"https://env.application.fr/agent/927/import-crep-papier\",
\"line-number\":762,\"original-policy\":\"default-src none; connect-src https://env.application.fr:443; font-src https://env.application.fr:443; img-src https://env.application.fr:443; script-src https://env.application.fr:443 unsafe-inline; style-src https://env.application.fr:443 unsafe-inline; report-uri https://serveur-csp.frcsp-cisirh.php\",
\"referrer\":\"https://env.application.fr/agent/927/import-crep-papier\",
\"script-sample\":\"call to eval() or related function blocked by CSP\",
\"source-file\":\"https://env.application.fr/js/a2d8d78.js\",
\"violated-directive\":\"script-src https://env.application.fr:443 unsafe-inline\"}}",
"type"=>"applicatif",
"csp-report"=>{"referrer"=>"https://env.application.fr/agent/927/import-crep-papier",
"script-sample"=>"call to eval() or related function blocked by CSP",
"original-policy"=>"default-src none; connect-src https://env.application.fr:443; font-src https://env.application.fr:443; img-src https://env.application.fr:443; script-src https://env.application.fr:443 unsafe-inline; style-src https://env.application.fr:443 unsafe-inline; report-uri https://serveur-csp.fr.php",
"source-file"=>"https://env.application.fr/js/a2d8d78.js",
"violated-directive"=>"script-src https://env.application.fr:443 unsafe-inline",
"document-uri"=>"https://env.application.fr/agent/927/import-crep-papier",
"line-number"=>762,
"blocked-uri"=>"self"},
uri"=>"https://env.application.fr/agent/927/import-crep-papier",
"tags"=>["json", "CSP", "_grokparsefailure"],
"@timestamp"=>2017-10-31T16:46:15.063Z, "@version"=>"1",
"beat"=>{"name"=>"hostname", "hostname"=>"hostname", "version"=>"5.6.1"},
"host"=>"hostname",
"fields"=>{"type"=>"CSP", "environment"=>"PRODJ"}}}
```
Merci pour votre aidre, j'y perds mon GROK ! 😅
David
EDIT : ajout de retour-chariots pour la lecture du JSON..
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 1, 2017, 6:14pm UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/2 "2017-11-01T18:14:49Z")
Idéalement, ça serait bien de faire un exemple beaucoup plus simple avec un document JSON contenant le texte que Grok doit lire et juste une config logstash comme:
```auto
input { stdin {} }
filter {
grok {
...
}
}
output { stdout { codec => rubydebug } }
```
puis
```auto
echo "YOUR CONTENT" | logstash -f logstash.conf
```
Ca permettrait de supprimer tout le bruit dans ton exemple.
Ensuite je te conseille de simplement utiliser quelque chose comme:
- [http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/)
- [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)
- Dans Kibana: [https://www.elastic.co/guide/en/kibana/current/grokdebugger-getting-started.html](https://www.elastic.co/guide/en/kibana/current/grokdebugger-getting-started.html)
---
### Author:  [@David\_ROELANDT](https://discuss.elastic.co/u/David_ROELANDT)
#### Post date: [November 2, 2017, 12:55pm UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/3 "2017-11-02T12:55:42Z")
Bonjour et merci pour ta réponse !
J'utilise Grok Debugger pour tester mes motifs, c'est un super outil. J'ai essayé d'utiliser l'outil présent dans Kibana mais j'ai tendance à bien faire planter Kibana avec.
Donc je résume :
1 - j'ai 1 fichier à parser avec X lignes formatées en JSON
2 - Le fichier est parcouru par filebeat
3 - Logstash reçoit la ligne JSON
3.1 - Split de la ligne json
`json{ source => "message"}`
3.2 récupération de la valeur document-uri
`mutate { add_field => { "uri" => "%{[csp-report][document-uri]}" } }`
3.3 Parsing du champ uri
`grok { match => { "uri" => "%{URI}"} }`
4 . envoi vers elasticsearch
Les étapes 1,2 , 3.1 , 3.2 et 4 fonctionnent parfaitement sans l'étape 3.3.
Le souci est au niveau du 3.3 (et impacte le 4 ensuite).
**Fichier 00-filter-json.conf**
```auto
input { stdin {} }
filter {
json{ source => "message"}
mutate { add_field => { "uri" => "%{[csp-report][document-uri]}" } }
grok { match => { "uri" => "%{URI}"} }
}
output { stdout { codec => rubydebug } }
```
### Voici la sortie (un peu formatée) :
**Chargement des filtres**
```auto
[2017-11-02T10:37:24,911][DEBUG][logstash.filters.grok] Adding pattern {"URIPROTO"=>"[A-Za-z]([A-Za-z0-9+\\-.]+)+"}
[2017-11-02T10:37:24,911][DEBUG][logstash.filters.grok] Adding pattern {"URIHOST"=>"%{IPORHOST}(?::%{POSINT:port})?"}
[2017-11-02T10:37:24,911][DEBUG][logstash.filters.grok] Adding pattern {"URIPATH"=>"(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%&_\\-]*)+"}
[2017-11-02T10:37:24,912][DEBUG][logstash.filters.grok] Adding pattern {"URIPARAM"=>"\\?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\\-\\[\\]<>]*"}
[2017-11-02T10:37:24,912][DEBUG][logstash.filters.grok] Adding pattern {"URIPATHPARAM"=>"%{URIPATH}(?:%{URIPARAM})?"}
[2017-11-02T10:37:24,912][DEBUG][logstash.filters.grok] Adding pattern {"URI"=>"%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?"}
```
```auto
[2017-11-02T10:37:25,124][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```
Entrée de l'event (3.1):
```auto
[2017-11-02T10:37:25,176][DEBUG][logstash.filters.json] Event after json filter {:event=>2017-11-02T09:37:25.092Z hostname {"csp-report":{"blocked-uri":"self","document-uri":"https://env.appli.orga/agent/927/import-crep-papier","line-number":762,"original-policy":"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php","referrer":"https://env.appli.orga/agent/927/import-crep-papier","script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.appli.orga/js/a2d8d78.js","violated-directive":"script-src https://env.appli.orga:443 unsafe-inline"}}}
```
Mutate add field uri (3.2):
```auto
[2017-11-02T10:37:25,179][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"uri", "value"=>["%{[csp-report][document-uri]}"]}
```
Grok (3.3)
```auto
[2017-11-02T10:37:25,180][DEBUG][logstash.filters.grok] Running grok filter {:event=>2017-11-02T09:37:25.092Z hostname {"csp-report":{"blocked-uri":"self","document-uri":"https://env.appli.orga/agent/927/import-crep-papier","line-number":762,"original-policy":"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php","referrer":"https://env.appli.orga/agent/927/import-crep-papier","script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.appli.orga/js/a2d8d78.js","violated-directive":"script-src https://env.appli.orga:443 unsafe-inline"}}}
```
Sortie (4)
```auto
[2017-11-02T10:37:25,182][DEBUG][logstash.filters.grok] Event now: {:event=>2017-11-02T09:37:25.092Z hostname {"csp-report":{"blocked-uri":"self","document-uri":"https://env.appli.orga/agent/927/import-crep-papier","line-number":762,"original-policy":"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php","referrer":"https://env.appli.orga/agent/927/import-crep-papier","script-sample":"call to eval() or related function blocked by CSP","source-file":"https://env.appli.orga/js/a2d8d78.js","violated-directive":"script-src https://env.appli.orga:443 unsafe-inline"}}}
```
```auto
[2017-11-02T10:37:25,183][DEBUG][logstash.pipeline] output received {"event"=>{"@timestamp"=>2017-11-02T09:37:25.092Z, "@version"=>"1", "host"=>"hostname", "message"=>"{\"csp-report\":{\"blocked-uri\":\"self\",\"document-uri\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"line-number\":762,\"original-policy\":\"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php\",\"referrer\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"script-sample\":\"call to eval() or related function blocked by CSP\",\"source-file\":\"https://env.appli.orga/js/a2d8d78.js\",\"violated-directive\":\"script-src https://env.appli.orga:443 unsafe-inline\"}}", "csp-report"=>{"referrer"=>"https://env.appli.orga/agent/927/import-crep-papier", "script-sample"=>"call to eval() or related function blocked by CSP", "original-policy"=>"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php", "source-file"=>"https://env.appli.orga/js/a2d8d78.js", "violated-directive"=>"script-src https://env.appli.orga:443 unsafe-inline", "document-uri"=>"https://env.appli.orga/agent/927/import-crep-papier", "line-number"=>762, "blocked-uri"=>"self"}, "uri"=>"https://env.appli.orga/agent/927/import-crep-papier"}}
```
---
### Author:  [@David\_ROELANDT](https://discuss.elastic.co/u/David_ROELANDT)
#### Post date: [November 2, 2017, 12:55pm UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/4 "2017-11-02T12:55:52Z")
**Output :**
```json
{
"@timestamp" => 2017-11-02T09:37:25.092Z,
"@version" => "1",
"host" => "hostname",
"message" => "{\"csp-report\":{\"blocked-uri\":\"self\",\"document-uri\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"line-number\":762,\"original-policy\":\"default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php\",\"referrer\":\"https://env.appli.orga/agent/927/import-crep-papier\",\"script-sample\":\"call to eval() or related function blocked by CSP\",\"source-file\":\"https://env.appli.orga/js/a2d8d78.js\",\"violated-directive\":\"script-src https://env.appli.orga:443 unsafe-inline\"}}",
"csp-report" => {
"referrer" => "https://env.appli.orga/agent/927/import-crep-papier",
"script-sample" => "call to eval() or related function blocked by CSP",
"original-policy" => "default-src none; connect-src https://env.appli.orga:443; font-src https://env.appli.orga:443; img-src https://env.appli.orga:443; script-src https://env.appli.orga:443 unsafe-inline; style-src https://env.appli.orga:443 unsafe-inline; report-uri https://appli.orga/csp.php",
"source-file" => "https://env.appli.orga/js/a2d8d78.js",
"violated-directive" => "script-src https://env.appli.orga:443 unsafe-inline",
"document-uri" => "https://env.appli.orga/agent/927/import-crep-papier",
"line-number" => 762,
"blocked-uri" => "self"
},
"uri" => "https://env.appli.orga/agent/927/import-crep-papier"
```
Arrêt de logstash
```auto
[2017-11-02T10:37:25,198][DEBUG][logstash.pipeline] Shutdown waiting for worker thread #
[2017-11-02T10:37:25,198][DEBUG][logstash.pipeline] Shutdown waiting for worker thread #
[2017-11-02T10:37:25,198][DEBUG][logstash.pipeline] Shutdown waiting for worker thread #
[2017-11-02T10:37:25,198][DEBUG][logstash.filters.json] closing {:plugin=>"LogStash::Filters::Json"}
[2017-11-02T10:37:25,199][DEBUG][logstash.filters.mutate] closing {:plugin=>"LogStash::Filters::Mutate"}
[2017-11-02T10:37:25,199][DEBUG][logstash.filters.grok] closing {:plugin=>"LogStash::Filters::Grok"}
[2017-11-02T10:37:25,385][DEBUG][logstash.outputs.stdout] closing {:plugin=>"LogStash::Outputs::Stdout"}
[2017-11-02T10:37:25,385][DEBUG][logstash.pipeline] Pipeline main has been shutdown
[2017-11-02T10:37:28,027][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-11-02T10:37:28,028][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-11-02T10:37:28,028][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-11-02T10:37:28,033][WARN][logstash.agent] stopping pipeline {:id=>"main"}
[2017-11-02T10:37:28,034][DEBUG][logstash.pipeline] Closing inputs
[2017-11-02T10:37:28,034][DEBUG][logstash.inputs.stdin] stopping {:plugin=>"LogStash::Inputs::Stdin"}
[2017-11-02T10:37:28,034][DEBUG][logstash.pipeline] Closed inputs
```
En fait, on dirait que GROK ne passe pas du tout sur le champ **"uri"** préalablement défini mais reste sur l'event global.. ?!
Au fur et à mesure de mes essais, j'ai pu voir que grok parsait le champ "message" en lieu et place du champ passé en paramètre : 🤨
Test de la configuration 🙂
```auto
[2017-11-02T13:46:49,445][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"grok", :type=>"filter", :class=>LogStash::Filters::Grok}
[2017-11-02T13:46:49,451][DEBUG][logstash.filters.grok] config LogStash::Filters::Grok/@match = {"uri"=>"%{URI}"}
```
```auto
[2017-11-02T13:47:11,863][DEBUG][logstash.filters.mutate] filters/LogStash::Filters::Mutate: removing field {:field=>"message"}
[2017-11-02T13:47:11,864][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"uri", "value"=>["%{[csp-report][document-uri]}"]}
[2017-11-02T13:47:11,865][DEBUG][logstash.filters.grok] Running grok filter {:event=>2017-11-02T12:47:11.738Z hostname %{message}}
[2017-11-02T13:47:11,868][DEBUG][logstash.filters.grok] Event now: {:event=>2017-11-02T12:47:11.738Z hostname %{message}}
```
J'ai donc cherché à remplacer le champ message par le contenu à parser :
```auto
input { stdin {} }
filter {
json { source => "message"}
mutate {remove_field => "message"}
mutate { add_field => { "message" => "%{[csp-report][document-uri]}" } }
grok { match => { "message" => "%{URI}"} }
}
output { stdout { codec => rubydebug } }
```
le résultat n'a pas été concluant. 😓
J'ai aussi essayé de d'accèder directement aux données du JSON :
`grok { match => { "%{[csp-report][document-uri]" => "%{URI}" } }`
ou
`grok { match => { "%{[message][csp-report][document-uri]" => "%{URI}" } }`
J'ai vraiment l'impression que le seul champ que Grok peut parser est le champ **"message"**
Je pense que je vais finir par utiliser insérer du code ruby pour m'en sortir..
David
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 6, 2017, 8:01am UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/5 "2017-11-06T08:01:32Z")
Peut-être que @colinsurprenant a une idée?
---
### Author:  [@David\_ROELANDT](https://discuss.elastic.co/u/David_ROELANDT)
#### Post date: [November 6, 2017, 8:47am UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/6 "2017-11-06T08:47:01Z")
Je n'ai pas précisé mais je suis en 5.6.1
J'ai fini par utiliser un bout de code ruby qui marche plutôt bien. 🙂
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 4, 2017, 8:47am UTC](https://discuss.elastic.co/t/logstash-csp-report-uri-non-parse/105930/7 "2017-12-04T08:47:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.