# Unable to parse CSP logs with filebeat

**URL:** <https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 1, 2024, 7:43am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279 "2024-07-01T07:43:59Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 7:43am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/1 "2024-07-01T07:43:59Z")

</div>

Hi Team,

I am trying to parse the csp logs with filebeat and I tried numerous option but none of them is working. Can someone please help?  
Here are the raw csp log

```auto
{"date":"01/Jul/2024:12:07:00 +0530", "IP address":"176.240.xxx.xxx", "http_x_forwarded_for":"", "status":"204", "http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/126.0.6478.153 Mobile/15E148 Safari/604.1", "body_bytes_sent":"0", "request":"POST /_csp_xx.xx.xx.xx HTTP/1.1","request_body": "{\"type\":\"csp-violation\",\"url\":\"https://www.xx.xx.xx.xx.com/\",\"csp-report\":{\"document-uri\":\"https://www.xx.xx.xx.xx.com/\",\"referrer\":\"https://www.google.com/\",\"violated-directive\":\"style-src-elem\",\"effective-directive\":\"style-src-elem\",\"original-policy\":\"default-src 'self' 'unsafe-inline'; frame-src 'self' td.doubleclick.net; manifest-src 'self' www.xx.xx.xx.xx.com; style-src 'self' 'unsafe-inline' *.googleapis.com 'self' fonts.googleapis.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' https://www.google-analytics.com 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-C8oQVJ33cKtnkARnmeWp6SDChkU+u7KvsNMFUzkkUzk=' 'sha256-TXAWuIqKdrNpSKYHYZ7bkGoIMlHGjhBceAOn7h5QUQw=' 'sha256-tqlMHUh+wHh08rh4nIZeMbwnBAcss9QVE8OKfdrvodU=' 'sha256-2b5RU9WZsUgm7tNV36A0w17RAyEyqDvuxTzvHVIQ6E0=' 'sha256-NZe6EI6DHYFUR8E7IB0jYeyXL+6P2HQzsVridcWVESE=' 'sha256-9l867tRreZQISfogIvLL3zaCiN6QRvKz2gdcQvnd6PY=' 'sha256-FXSlRA54YYcbC4EqhWvLY4A55v1v4ONi1Rk2lHMJZf8=' 'sha256-8ijKzAGJMbNBQIOqLZ3pM/92KPYjh08Mm/QzYPSJ2e8=' 'sha256-WMOEx/fVLpdP2x5+htiQ6TwHSKlY31r0FFIbtbfIfIM=' ajax.googleapis.com https://www.xx.xx.xx.xx.com; script-src-attr 'self'; style-src-attr 'self' 'unsafe-inline'; script-src-elem 'self' 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' https://www.googletagmanager.com http://fonts.googleapis.com https://www.google-analytics.com https://ajax.googleapis.com; style-src-elem 'self' 'unsafe-inline' www.xx.xx.xx.xx.com gc.kis.v2.scr.kaspersky-labs.comi http://fonts.googleapis.com; media-src 'self' data:; connect-src 'self' analytics.google.com stats.g.doubleclick.net www.google-analytics.com www.google.com.sg www.google.co.in region1.analytics.google.com region1.google-analytics.com www.google.co.jp www.google.co.th www.google.nl www.google.fr www.google.com.ng www.google.ae www.google.com.hk; font-src 'self' www.xx.xx.xx.xx.com http://fonts.gstatic.com http://fonts.gstatic.com data:; img-src 'self' www.xx.xx.xx.xx.com www.google.co.in www.google.com www.google-analytics.com www.google.co.uk www.googletagmanager.com www.google.co.jp www.google.ru www.google.co.th www.google.com.om analytics.google.com stats.g.doubleclick.net www.google.co.kr www.google.cz www.google.nl www.google.fr www.google.com.sa www.google.com.bh data: www.google.com.mx www.google.com.au www.google.com.ng www.google.com.sg www.google.ae www.google.de www.google.com.hk; object-src 'none'; base-uri 'self'; upgrade-insecure-requests; report-uri https://csp.xx.xx.xx.xx.in/_csp_xx.xx.xx.xx\",\"blocked-uri\":\"http://fonts.googleapis.com/css?family=Montserrat:400,700\",\"status-code\":0}}"}

```

This is my filbeat parser

```auto
filebeat.inputs.0:
- type: log
  enabled: true
  paths:
    - /var/log/csp.log
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
  json.keys_under_root: true
  json.add_error_key: true
processors:
  - decode_json_fields:
      fields: ["message"]

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

```

I even tried adjusting json pipeline

```auto
curl -k -u 'elastic:xxxxxx' -XPUT "https://10.122.xx.xx:9200/_ingest/pipeline/json_pipeline" -H "Content-Type: application/json" -d '{
  "description": "Pipeline to parse nested request_body field",
  "processors": [
    {
      "json": {
        "field": "request_body",
        "target_field": "request_body",
        "add_to_root": false
      }
    }
  ]
}'

```

---

<div class="post-metadata">

**Author:** ![Alex\_Salgado-Elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_salgado-elastic/32/103081_2.png) [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)\
**Post date:** [July 1, 2024, 10:25am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/2 "2024-07-01T10:25:46Z")

</div>

> [@Blason](#):
>
> ```auto
> filebeat.inputs.0:
> 
> ```

Have you checked the following points?

- [Filebeat Input Log Documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html)
- [Filebeat Multiline Examples](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)
- [Filebeat Decode JSON Fields Documentation](https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html)

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 11:41am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/3 "2024-07-01T11:41:59Z")

</div>

Yes - I read through those already but not sure if I am missing anything?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 1, 2024, 12:07pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/4 "2024-07-01T12:07:20Z")

</div>

> [@Blason](#):
>
> Yes - I read through those already but not sure if I am missing anything?

Check the first link, your configuration should be `filebeat.inputs`, not `filebeat.inputs.0`.

Have you tried using this? :

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp.log
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
  json.keys_under_root: true
  json.add_error_key: true

```

Also, is your source file a multiline file or does it have a json document per line?

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 12:13pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/5 "2024-07-01T12:13:55Z")

</div>

I spent so many days with that that

```auto
filebeat.inputs

```

And it did not work - while one of the google search pointed me like that and filebeat started clean.  
And json is single line and not multiline - The multiline I kept it so that I was doing lot of R&D to get those stuff parsed ☹

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 12:16pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/6 "2024-07-01T12:16:29Z")

</div>

OK here is my modified one

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp2.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: "message"
  #processors:
  # - decode_json_fields:
  # fields: ["message"]

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

```

However still no luck

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 1, 2024, 12:25pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/7 "2024-07-01T12:25:47Z")

</div>

What do you have in filebeat logs? Also, share your entire `filebeat.yml`

You need to troubleshoot this step by step, first you need to make it arrive into Elasticsearch, then you start dealing with the parsing.

Also, where you want to parse it? In Filebeat or in Elasticsearch? Because you also have an ingest pipeline.

I would say that it would be easier to use filebeat just to send the logs and do all the parsing in Elasticsearch.

> [@Blason](#):
>
> And json is single line and not multiline

The `csp2.log` is a single line json?

What is the result of the command `cat /var/log/csp2.log | wc -l` on your server?

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 12:42pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/8 "2024-07-01T12:42:02Z")

</div>

```auto
cat /var/log/csp2.log | wc -l
1

```

And here is filebeat file

```auto
cat /etc/filebeat/filebeat.yml | grep -v "#" | grep -vE '^
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp2.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: "message"
setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /tmp/ir.json
  json.keys_under_root: true
  json.add_error_key: true
  json.expand_keys: true
  json.overwrite_keys: true
processors:
  - decode_json_fields:
      fields: ["message"]
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["10.122.xxx.xx:9200"]
  ssl.certificate_authorities: ["/ca.crt"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"
  pipeline: "json_pipeline"
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 12:42pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/9 "2024-07-01T12:42:55Z")

</div>

And ofcourse even I am looking to parse it in filebeat instead of elasticsearch. However since I am running out of option I am trying whatever I am aware of

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 1, 2024, 12:56pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/10 "2024-07-01T12:56:55Z")

</div>

> [@Blason](#):
>
> However since I am running out of option I am trying whatever I am aware of

Yeah, but to troubleshoot things you may need to take a step back to try to see what is happening and where is the error.

You didn't share the filebeat logs, please share the logs as well.

Also, your `filebeat.yml` has 3 `filebeat.inputs`, you should have just on `filebeat.inputs` and in then you configure each input (the `type` setting) as the example on the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html).

I'm not sure how filebeat will work when you have multiple `filebeat.inputs`, I would assume that it would ignore the duplicates and use only the last one.

It seems that you have multiple lines duplicated, I'm also not sure how filebeat behaves in this situation.

You have 2 files in the configuration, `ir.json` and the `csp2.log`, are both of them not working?

I would take a step back and use this simplified `filebeat.yml`

```auto
filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/csp2.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["10.122.xxx.xx:9200"]
  ssl.certificate_authorities: ["/ca.crt"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

Then, after changing the `filebeat.yml` test both the config and the output.

Use `filebeat test config` and `filebeat test output` and share the results.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 1:02pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/11 "2024-07-01T13:02:34Z")

</div>

> [@leandrojmp](#):
>
> filebeat test output

```auto
elasticsearch: https://10.122.0.11:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.122.0.11
    dial up... OK
  TLS...
    security... WARN server's certificate chain verification is disabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.17.4

```

And here is

```auto
 filebeat test config
Config OK

```

```auto

2024-07-01T18:31:39.359+0530 INFO instance/beat.go:685 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2024-07-01T18:31:39.359+0530 INFO instance/beat.go:693 Beat ID: fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca
2024-07-01T18:31:39.362+0530 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2024-07-01T18:31:39.362+0530 INFO [beat] instance/beat.go:1039 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca"}}}
2024-07-01T18:31:39.362+0530 INFO [beat] instance/beat.go:1048 Build info {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2024-07-01T18:31:39.362+0530 INFO [beat] instance/beat.go:1051 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.9"}}}
2024-07-01T18:31:39.363+0530 INFO [beat] instance/beat.go:1055 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-11-30T13:03:55+05:30","containerized":false,"name":"isn-siem","ip":["127.0.0.1/8","::1/128","139.59.15.218/20","10.47.0.14/16","fe80::a83c:ebff:fea9:d284/64","10.122.0.11/20","fe80::9816:98ff:fe04:3e7c/64"],"kernel_version":"5.4.0-167-generic","mac":["aa:3c:eb:a9:d2:84","9a:16:98:04:3e:7c"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"IST","timezone_offset_sec":19800,"id":"1a0aa414926b27af0d1e73d662a01ab4"}}}
2024-07-01T18:31:39.363+0530 INFO [beat] instance/beat.go:1084 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 2884214, "ppid": 2881357, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2024-07-01T18:31:39.200+0530"}}}
2024-07-01T18:31:39.363+0530 INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.4
2024-07-01T18:31:39.363+0530 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.17.4' as ILM is enabled.
2024-07-01T18:31:39.364+0530 WARN [cfgwarn] tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2024-07-01T18:31:39.364+0530 INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.122.0.11:9200
2024-07-01T18:31:39.364+0530 INFO [publisher] pipeline/module.go:113 Beat name: isn-siem
2024-07-01T18:31:39.366+0530 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2024-07-01T18:31:39.366+0530 INFO instance/beat.go:492 filebeat start running.
2024-07-01T18:31:39.367+0530 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=43175470
2024-07-01T18:31:39.446+0530 INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=43186697
2024-07-01T18:31:39.447+0530 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 2
2024-07-01T18:31:39.447+0530 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2024-07-01T18:31:39.447+0530 INFO [crawler] beater/crawler.go:117 starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.paths.0 filebeat.inputs.0.type]
2024-07-01T18:31:39.447+0530 ERROR [input] input-logfile/manager.go:176 filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat
2024-07-01T18:31:39.447+0530 INFO [crawler] beater/crawler.go:148 Starting input (ID: 17228493714276037636)
2024-07-01T18:31:39.447+0530 INFO [input.filestream] compat/compat.go:111 Input filestream starting {"id": "EF17E7C0AE793404"}
2024-07-01T18:31:39.448+0530 INFO [file_watcher] filestream/fswatch.go:140 Start next scan
2024-07-01T18:31:39.454+0530 INFO [crawler] beater/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2024-07-01T18:31:39.454+0530 INFO cfgfile/reload.go:164 Config reloader started
2024-07-01T18:31:39.464+0530 INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.122.0.11:9200
2024-07-01T18:31:39.470+0530 INFO [esclientleg] eslegclient/connection.go:285 Attempting to connect to Elasticsearch version 7.17.4
2024-07-01T18:31:39.475+0530 INFO cfgfile/reload.go:224 Loading of config files completed.
2024-07-01T18:31:39.475+0530 INFO [input.httpjson-cursor] compat/compat.go:111 Input httpjson-cursor starting {"id": "F28AEA1431B7F788"}
2024-07-01T18:31:39.475+0530 INFO [input.httpjson-stateless] compat/compat.go:111 Input httpjson-stateless starting {"id": "A3B2CB572CE313E5"}
2024-07-01T18:31:39.475+0530 INFO [input.httpjson-stateless] compat/compat.go:111 Input httpjson-stateless starting {"id": "86F57B6043A0C821"}
2024-07-01T18:31:39.475+0530 INFO [input.httpjson-stateless] compat/compat.go:111 Input httpjson-stateless starting {"id": "F8929C8C8CD1AAC0"}
2024-07-01T18:31:39.475+0530 INFO [input.httpjson-stateless] v2/input.go:112 Process another repeated request. {"id": "A3B2CB572CE313E5", "input_url": "https://urlhaus-api.abuse.ch/v1/urls/recent/"}
2024-07-01T18:31:39.475+0530 INFO [input.httpjson-stateless] v2/input.go:112 Process another repeated request. {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2024-07-01T18:31:39.476+0530 INFO [input.httpjson-stateless] v2/input.go:112 Process another repeated request. {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2024-07-01T18:31:39.476+0530 INFO [input.httpjson-cursor] v2/input.go:112 Process another repeated request. {"id": "F28AEA1431B7F788", "input_source": "https://otx.alienvault.com/api/v1/indicators/export", "input_url": "https://otx.alienvault.com/api/v1/indicators/export"}
2024-07-01T18:31:40.448+0530 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://10.122.0.11:9200))
2024-07-01T18:31:40.449+0530 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2024-07-01T18:31:40.449+0530 INFO [publisher] pipeline/retry.go:223 done
2024-07-01T18:31:40.454+0530 INFO [esclientleg] eslegclient/connection.go:285 Attempting to connect to Elasticsearch version 7.17.4
2024-07-01T18:31:40.471+0530 INFO [esclientleg] eslegclient/connection.go:285 Attempting to connect to Elasticsearch version 7.17.4
2024-07-01T18:31:40.485+0530 INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2024-07-01T18:31:40.492+0530 INFO [index-management.ilm] ilm/std.go:170 ILM policy filebeat exists already.
2024-07-01T18:31:40.492+0530 INFO [index-management] idxmgmt/std.go:397 Set setup.template.name to '{filebeat-7.17.4 {now/d}-000001}' as ILM is enabled.
2024-07-01T18:31:40.492+0530 INFO [index-management] idxmgmt/std.go:402 Set setup.template.pattern to 'filebeat-7.17.4-*' as ILM is enabled.
2024-07-01T18:31:40.492+0530 INFO [index-management] idxmgmt/std.go:436 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.4 {now/d}-000001} as ILM is enabled.
2024-07-01T18:31:40.492+0530 INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2024-07-01T18:31:40.509+0530 INFO template/load.go:110 Template "filebeat-7.17.4" already exists and will not be overwritten.
2024-07-01T18:31:40.509+0530 INFO [index-management] idxmgmt/std.go:297 Loaded index template.
2024-07-01T18:31:40.512+0530 INFO [index-management.ilm] ilm/std.go:126 Index Alias filebeat-7.17.4 exists already.
2024-07-01T18:31:40.519+0530 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://10.122.0.11:9200)) established
2024-07-01T18:31:40.935+0530 INFO [input.httpjson-stateless] v2/request.go:204 request finished: 1000 events published {"id": "A3B2CB572CE313E5", "input_url": "https://urlhaus-api.abuse.ch/v1/urls/recent/"}
2024-07-01T18:31:42.167+0530 INFO [input.httpjson-stateless] v2/request.go:204 request finished: 48 events published {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2024-07-01T18:31:42.816+0530 WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.July, 1, 18, 31, 42, 164873675, time.Local), Meta:{"_id":"a9babe52240c6472986cc2e8f58ef8aa37cabe307fac15101569cc5ab953e8ef","pipeline":"filebeat-7.17.4-threatintel-malwarebazaar-pipeline"}, Fields:{"agent":{"ephemeral_id":"52d736a5-5ed2-40f5-b709-742488c324f4","hostname":"isn-siem","id":"fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca","name":"isn-siem","type":"filebeat","version":"7.17.4"},"ecs":{"version":"1.12.0"},"event":{"created":"2024-07-01T13:01:42.164Z","dataset":"threatintel.malwarebazaar","module":"threatintel"},"fileset":{"name":"malwarebazaar"},"input":{"type":"httpjson"},"json":{"anonymous":0,"code_sign":[{"algorithm":"sha256WithRSAEncryption","issuer_cn":"Fjeldskreddets","serial_number":"55432c16faecc698c83e18bd4edfa7b279218f7b","subject_cn":"Fjeldskreddets","thumbprint":"a98702daf3f959c9f17852d76641ee02cfab306351b8b8539c98780b2cc21565","thumbprint_algorithm":"SHA256","valid_from":"2023-10-17T07:04:51Z","valid_to":"2026-10-16T07:04:51Z"}],"dhash_icon":"a5a026f1e0f2f230","file_name":"SeAH RFP_24-0676·pdf.exe","file_size":486976,"file_type":"exe","file_type_mime":"application/x-dosexec","first_seen":"2024-07-01 12:50:01","gimphash":null,"imphash":"b40f29cd171eb54c01b1dd2683c9c26b","intelligence":{"clamav":null,"downloads":"232","mail":null,"uploads":"1"},"last_seen":null,"md5_hash":"8951c491b26675b308464af7a29567bd","origin_country":"HU","reporter":"adrian__luca","sha1_hash":"883bd6d014e3baf9141b304519fc34eef20eb41f","sha256_hash":"3fc5e4d002e04269f2f674e6a2e98935df133ffe0f1fd54c817662d864c2f1b8","sha3_384_hash":"edf5e838b61c9948da7ea4a348eba43ac482625013c8053f6a7126b2143a08e4279b89afaab73f69c927f92b6dca70e6","signature":"Loki","ssdeep":"12288:/qgowhL+Pylw1QeQMQukEsvs5uT8JGjD8WpyLk8n:xR+a3M0ee8JG38lr","tags":["exe","Loki","signed"],"telfhash":null,"tlsh":"T1FAA4F187F6049076E51DACF34B6FC66FA92B6F40363A0603D7D07A2A067D6F66B13046"},"message":"{\"anonymous\":0,\"code_sign\":[{\"algorithm\":\"sha256WithRSAEncryption\",\"issuer_cn\":\"Fjeldskreddets\",\"serial_number\":\"55432c16faecc698c83e18bd4edfa7b279218f7b\",\"subject_cn\":\"Fjeldskreddets\",\"thumbprint\":\"a98702daf3f959c9f17852d76641ee02cfab306351b8b8539c98780b2cc21565\",\"thumbprint_algorithm\":\"SHA256\",\"valid_from\":\"2023-10-17T07:04:51Z\",\"valid_to\":\"2026-10-16T07:04:51Z\"}],\"dhash_icon\":\"a5a026f1e0f2f230\",\"file_name\":\"SeAH RFP_24-0676·pdf.exe\",\"file_size\":486976,\"file_type\":\"exe\",\"file_type_mime\":\"application/x-dosexec\",\"first_seen\":\"2024-07-01 12:50:01\",\"gimphash\":null,\"imphash\":\"b40f29cd171eb54c01b1dd2683c9c26b\",\"intelligence\":{\"clamav\":null,\"downloads\":\"232\",\"mail\":null,\"uploads\":\"1\"},\"last_seen\":null,\"md5_hash\":\"8951c491b26675b308464af7a29567bd\",\"origin_country\":\"HU\",\"reporter\":\"adrian__luca\",\"sha1_hash\":\"883bd6d014e3baf9141b304519fc34eef20eb41f\",\"sha256_hash\":\"3fc5e4d002e04269f2f674e6a2e98935df133ffe0f1fd54c817662d864c2f1b8\",\"sha3_384_hash\":\"edf5e838b61c9948da7ea4a348eba43ac482625013c8053f6a7126b2143a08e4279b89afaab73f69c927f92b6dca70e6\",\"signature\":\"Loki\",\"ssdeep\":\"12288:/qgowhL+Pylw1QeQMQukEsvs5uT8JGjD8WpyLk8n:xR+a3M0ee8JG38lr\",\"tags\":[\"exe\",\"Loki\",\"signed\"],\"telfhash\":null,\"tlsh\":\"T1FAA4F187F6049076E51DACF34B6FC66FA92B6F40363A0603D7D07A2A067D6F66B13046\"}","service":{"type":"threatintel"},"tags":["threatintel-malwarebazaar","forwarded"]}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [threatintel.malwarebazaar.code_sign] of type [keyword] in document with id 'a9babe52240c6472986cc2e8f58ef8aa37cabe307fac15101569cc5ab953e8ef'. Preview of field's value: '{subject_cn=Fjeldskreddets, issuer_cn=Fjeldskreddets, valid_to=2026-10-16T07:04:51Z, thumbprint=a98702daf3f959c9f17852d76641ee02cfab306351b8b8539c98780b2cc21565, valid_from=2023-10-17T07:04:51Z, serial_number=55432c16faecc698c83e18bd4edfa7b279218f7b, thumbprint_algorithm=SHA256, algorithm=sha256WithRSAEncryption}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:2511"}}, dropping event!

```

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 1, 2024, 1:06pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/12 "2024-07-01T13:06:06Z")

</div>

After this - Json messages does not get parsed and all those messages gets in message field

```auto
{
  "_index": "filebeat-7.17.4-2024.06.27-000105",
  "_type": "_doc",
  "_id": "McxjbpAB0yAvttXtAX0o",
  "_version": 1,
  "_score": 1,
  "_source": {
    "@timestamp": "2024-07-01T13:01:39.448Z",
    "ecs": {
      "version": "1.12.0"
    },
    "host": {
      "id": "1a0aa414926b27af0d1e73d662a01ab4",
      "containerized": false,
      "ip": [
        "10.47.0.14",
        "fe80::a83c:ebff:fea9:d284",
        "10.122.0.11",
        "fe80::9816:98ff:fe04:3e7c"
      ],
      "mac": [
        "aa:3c:eb:a9:d2:84",
        "9a:16:98:04:3e:7c"
      ],
      "hostname": "isn-siem",
      "architecture": "x86_64",
      "name": "isn-siem",
      "os": {
        "name": "Ubuntu",
        "kernel": "5.4.0-167-generic",
        "codename": "focal",
        "type": "linux",
        "platform": "ubuntu",
        "version": "20.04.4 LTS (Focal Fossa)",
        "family": "debian"
      }
    },
    "agent": {
      "hostname": "isn-siem",
      "ephemeral_id": "52d736a5-5ed2-40f5-b709-742488c324f4",
      "id": "fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca",
      "name": "isn-siem",
      "type": "filebeat",
      "version": "7.17.4"
    },
    "log": {
      "file": {
        "path": "/var/log/csp2.log"
      },
      "offset": 0
    },
    "message": "{\"date\":\"01/Jul/2024:11:50:43 +0530\",\"IP address\":\"49.xx.xx.xx\",\"http_x_forwarded_for\":\"\",\"status\":\"204\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15\",\"body_bytes_sent\":\"0\",\"request\":\"POST /_csp_xx.xx.xx.xx HTTP/1.1\",\"request_body\":\"{\\\"type\\\":\\\"csp-violation\\\",\\\"url\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"csp-report\\\":{\\\"document-uri\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"referrer\\\":\\\"https://www.google.com/\\\",\\\"violated-directive\\\":\\\"style-src-elem\\\",\\\"effective-directive\\\":\\\"style-src-elem\\\",\\\"original-policy\\\":\\\"default-src 'self' 'unsafe-inline'; frame-src 'self' td.doubleclick.net; manifest-src 'self' www.xx.xx.xx.xx.com; style-src 'self' 'unsafe-inline' *.googleapis.com 'self' fonts.googleapis.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' https://www.google-analytics.com 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-C8oQVJ33cKtnkARnmeWp6SDChkU+u7KvsNMFUzkkUzk=' 'sha256-TXAWuIqKdrNpSKYHYZ7bkGoIMlHGjhBceAOn7h5QUQw=' 'sha256-tqlMHUh+wHh08rh4nIZeMbwnBAcss9QVE8OKfdrvodU=' 'sha256-2b5RU9WZsUgm7tNV36A0w17RAyEyqDvuxTzvHVIQ6E0=' 'sha256-NZe6EI6DHYFUR8E7IB0jYeyXL+6P2HQzsVridcWVESE=' 'sha256-9l867tRreZQISfogIvLL3zaCiN6QRvKz2gdcQvnd6PY=' 'sha256-FXSlRA54YYcbC4EqhWvLY4A55v1v4ONi1Rk2lHMJZf8=' 'sha256-8ijKzAGJMbNBQIOqLZ3pM/92KPYjh08Mm/QzYPSJ2e8=' 'sha256-WMOEx/fVLpdP2x5+htiQ6TwHSKlY31r0FFIbtbfIfIM=' ajax.googleapis.com https://www.xx.xx.xx.xx.com; script-src-attr 'self'; style-src-attr 'self' 'unsafe-inline'; script-src-elem 'self' 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' https://www.googletagmanager.com http://fonts.googleapis.com https://www.google-analytics.com https://ajax.googleapis.com; style-src-elem 'self' 'unsafe-inline' www.xx.xx.xx.xx.com gc.kis.v2.scr.kaspersky-labs.comi http://fonts.googleapis.com; media-src 'self' data:; connect-src 'self' analytics.google.com stats.g.doubleclick.net www.google-analytics.com www.google.com.sg www.google.co.in region1.analytics.google.com region1.google-analytics.com www.google.co.jp www.google.co.th www.google.nl www.google.fr www.google.com.ng www.google.ae www.google.com.hk; font-src 'self' www.xx.xx.xx.xx.com http://fonts.gstatic.com http://fonts.gstatic.com data:; img-src 'self' www.xx.xx.xx.xx.com www.google.co.in www.google.com www.google-analytics.com www.google.co.uk www.googletagmanager.com www.google.co.jp www.google.ru www.google.co.th www.google.com.om analytics.google.com stats.g.doubleclick.net www.google.co.kr www.google.cz www.google.nl www.google.fr www.google.com.sa www.google.com.bh data: www.google.com.mx www.google.com.au www.google.com.ng www.google.com.sg www.google.ae www.google.de www.google.com.hk; object-src 'none'; base-uri 'self'; upgrade-insecure-requests; report-uri https://csp.xx.xx.xx.xx/_csp_xx.xx.xx.xx\\\",\\\"blocked-uri\\\":\\\"http://fonts.googleapis.com/css?family=Open+Sans:400,300\\\",\\\"status-code\\\":0}}\"}",
    "input": {
      "type": "filestream"
    }
  },
  "fields": {
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "is-siem"
    ],
    "host.mac": [
      "aa:3c:eb:a9:d2:84",
      "9a:16:98:04:3e:7c"
    ],
    "host.ip": [
      ,
      "10.47.0.14",
      "fe80::a83c:ebff:fea9:d284",
      "10.122.xx.xx",
      "fe80::9816:98ff:fe04:3e7c"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.version": [
      "20.04.4 LTS (Focal Fossa)"
    ],
    "host.os.kernel": [
      "5.4.0-167-generic"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "isn-siem"
    ],
    "host.name": [
      "isn-siem"
    ],
    "host.id": [
      "1a0aa414926b27af0d1e73d662a01ab4"
    ],
    "host.os.type": [
      "linux"
    ],
    "host.os.codename": [
      "focal"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      0
    ],
    "agent.hostname": [
      "isn-siem"
    ],
    "message": [
      "{\"date\":\"01/Jul/2024:11:50:43 +0530\",\"IP address\":\"49.37.8.152\",\"http_x_forwarded_for\":\"\",\"status\":\"204\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15\",\"body_bytes_sent\":\"0\",\"request\":\"POST /_csp_xx.xx.xx.xx HTTP/1.1\",\"request_body\":\"{\\\"type\\\":\\\"csp-violation\\\",\\\"url\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"csp-report\\\":{\\\"document-uri\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"referrer\\\":\\\"https://www.google.com/\\\",\\\"violated-directive\\\":\\\"style-src-elem\\\",\\\"effective-directive\\\":\\\"style-src-elem\\\",\\\"original-policy\\\":\\\"default-src 'self' 'unsafe-inline'; frame-src 'self' td.doubleclick.net; manifest-src 'self' www.xx.xx.xx.xx.com; style-src 'self' 'unsafe-inline' *.googleapis.com 'self' fonts.googleapis.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' https://www.google-analytics.com 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-C8oQVJ33cKtnkARnmeWp6SDChkU+u7KvsNMFUzkkUzk=' 'sha256-TXAWuIqKdrNpSKYHYZ7bkGoIMlHGjhBceAOn7h5QUQw=' 'sha256-tqlMHUh+wHh08rh4nIZeMbwnBAcss9QVE8OKfdrvodU=' 'sha256-2b5RU9WZsUgm7tNV36A0w17RAyEyqDvuxTzvHVIQ6E0=' 'sha256-NZe6EI6DHYFUR8E7IB0jYeyXL+6P2HQzsVridcWVESE=' 'sha256-9l867tRreZQISfogIvLL3zaCiN6QRvKz2gdcQvnd6PY=' 'sha256-FXSlRA54YYcbC4EqhWvLY4A55v1v4ONi1Rk2lHMJZf8=' 'sha256-8ijKzAGJMbNBQIOqLZ3pM/92KPYjh08Mm/QzYPSJ2e8=' 'sha256-WMOEx/fVLpdP2x5+htiQ6TwHSKlY31r0FFIbtbfIfIM=' ajax.googleapis.com https://www.xx.xx.xx.xx.com; script-src-attr 'self'; style-src-attr 'self' 'unsafe-inline'; script-src-elem 'self' 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' https://www.googletagmanager.com http://fonts.googleapis.com https://www.google-analytics.com https://ajax.googleapis.com; style-src-elem 'self' 'unsafe-inline' www.xx.xx.xx.xx.com gc.kis.v2.scr.kaspersky-labs.comi http://fonts.googleapis.com; media-src 'self' data:; connect-src 'self' analytics.google.com stats.g.doubleclick.net www.google-analytics.com www.google.com.sg www.google.co.in region1.analytics.google.com region1.google-analytics.com www.google.co.jp www.google.co.th www.google.nl www.google.fr www.google.com.ng www.google.ae www.google.com.hk; font-src 'self' www.xx.xx.xx.xx.com http://fonts.gstatic.com http://fonts.gstatic.com data:; img-src 'self' www.xx.xx.xx.xx.com www.google.co.in www.google.com www.google-analytics.com www.google.co.uk www.googletagmanager.com www.google.co.jp www.google.ru www.google.co.th www.google.com.om analytics.google.com stats.g.doubleclick.net www.google.co.kr www.google.cz www.google.nl www.google.fr www.google.com.sa www.google.com.bh data: www.google.com.mx www.google.com.au www.google.com.ng www.google.com.sg www.google.ae www.google.de www.google.com.hk; object-src 'none'; base-uri 'self'; upgrade-insecure-requests; report-uri https://csp.xx.xx.xx.xx/_csp_xx.xx.xx.xx\\\",\\\"blocked-uri\\\":\\\"http://fonts.googleapis.com/css?family=Open+Sans:400,300\\\",\\\"status-code\\\":0}}\"}"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "@timestamp": [
      "2024-07-01T13:01:39.448Z"
    ],
    "agent.id": [
      "fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "host.containerized": [
      false
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.file.path": [
      "/var/log/csp2.log"
    ],
    "agent.ephemeral_id": [
      "52d736a5-5ed2-40f5-b709-742488c324f4"
    ],
    "agent.version": [
      "7.17.4"
    ],
    "host.os.family": [
      "debian"
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 2, 2024, 6:29am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/13 "2024-07-02T06:29:35Z")

</div>

Still no luck ☹

Any ideas folks - can someone please help me in this configuration?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 2, 2024, 1:49pm UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/14 "2024-07-02T13:49:59Z")

</div>

What is the `filebeat.yml` that you are using now? The one I shared?

Since your messages are arriving at Elasticsearch, you now can try adding the processor in filebeat to parse it.

Try adding this to your `filebeat.yml`

```auto
processors:
  - decode_json_fields:
      fields: "message"
      max_depth: 2
      target: "parsed_json"
      overwrite_keys: false
      add_error_key: true

```

This would parse your `message` field, up to 2 nested levels, and store the result in a field named `parsed_json`.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 3, 2024, 2:55am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/15 "2024-07-03T02:55:37Z")

</div>

Yes I am using your json and let me try adding that field.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 3, 2024, 3:12am UTC](https://discuss.elastic.co/t/unable-to-parse-csp-logs-with-filebeat/362279/16 "2024-07-03T03:12:34Z")

</div>

> [@leandrojmp](#):
>
> `parsed_json`

Now this is my error

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/5/053d93b6db1f3504c2ee6751e7207e8bfdd58602.png)

And here is my file

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp2.log
      #json.keys_under_root: true
  json.add_error_key: true
  # json.message_key: "request_body"
processors:
  - decode_json_fields:
      fields: "message"
      max_depth: 2
      target: "parsed_json"
      overwrite_keys: false
      add_error_key: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["10.122.0.11:9200"]
  ssl.certificate_authorities: ["/ca.crt"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```