Unable to parse CSP logs with filebeat

Blason · July 1, 2024, 7:43am

Hi Team,

I am trying to parse the csp logs with filebeat and I tried numerous option but none of them is working. Can someone please help?
Here are the raw csp log

{"date":"01/Jul/2024:12:07:00 +0530", "IP address":"176.240.xxx.xxx", "http_x_forwarded_for":"", "status":"204", "http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/126.0.6478.153 Mobile/15E148 Safari/604.1", "body_bytes_sent":"0", "request":"POST /_csp_xx.xx.xx.xx HTTP/1.1","request_body": "{\"type\":\"csp-violation\",\"url\":\"https://www.xx.xx.xx.xx.com/\",\"csp-report\":{\"document-uri\":\"https://www.xx.xx.xx.xx.com/\",\"referrer\":\"https://www.google.com/\",\"violated-directive\":\"style-src-elem\",\"effective-directive\":\"style-src-elem\",\"original-policy\":\"default-src 'self' 'unsafe-inline'; frame-src 'self' td.doubleclick.net;  manifest-src 'self' www.xx.xx.xx.xx.com;  style-src 'self' 'unsafe-inline' *.googleapis.com 'self' fonts.googleapis.com;  script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' https://www.google-analytics.com 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-C8oQVJ33cKtnkARnmeWp6SDChkU+u7KvsNMFUzkkUzk=' 'sha256-TXAWuIqKdrNpSKYHYZ7bkGoIMlHGjhBceAOn7h5QUQw=' 'sha256-tqlMHUh+wHh08rh4nIZeMbwnBAcss9QVE8OKfdrvodU=' 'sha256-2b5RU9WZsUgm7tNV36A0w17RAyEyqDvuxTzvHVIQ6E0=' 'sha256-NZe6EI6DHYFUR8E7IB0jYeyXL+6P2HQzsVridcWVESE=' 'sha256-9l867tRreZQISfogIvLL3zaCiN6QRvKz2gdcQvnd6PY=' 'sha256-FXSlRA54YYcbC4EqhWvLY4A55v1v4ONi1Rk2lHMJZf8=' 'sha256-8ijKzAGJMbNBQIOqLZ3pM/92KPYjh08Mm/QzYPSJ2e8=' 'sha256-WMOEx/fVLpdP2x5+htiQ6TwHSKlY31r0FFIbtbfIfIM=' ajax.googleapis.com https://www.xx.xx.xx.xx.com;  script-src-attr 'self';  style-src-attr 'self' 'unsafe-inline';  script-src-elem 'self' 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' https://www.googletagmanager.com http://fonts.googleapis.com https://www.google-analytics.com https://ajax.googleapis.com;  style-src-elem 'self' 'unsafe-inline' www.xx.xx.xx.xx.com gc.kis.v2.scr.kaspersky-labs.comi http://fonts.googleapis.com;  media-src 'self' data:;  connect-src 'self' analytics.google.com stats.g.doubleclick.net www.google-analytics.com www.google.com.sg www.google.co.in region1.analytics.google.com region1.google-analytics.com www.google.co.jp www.google.co.th www.google.nl www.google.fr www.google.com.ng www.google.ae www.google.com.hk;  font-src 'self' www.xx.xx.xx.xx.com http://fonts.gstatic.com http://fonts.gstatic.com data:;  img-src 'self' www.xx.xx.xx.xx.com www.google.co.in www.google.com www.google-analytics.com www.google.co.uk www.googletagmanager.com www.google.co.jp www.google.ru www.google.co.th www.google.com.om analytics.google.com stats.g.doubleclick.net www.google.co.kr www.google.cz www.google.nl www.google.fr www.google.com.sa www.google.com.bh data: www.google.com.mx www.google.com.au www.google.com.ng www.google.com.sg www.google.ae www.google.de www.google.com.hk; object-src 'none'; base-uri 'self'; upgrade-insecure-requests; report-uri https://csp.xx.xx.xx.xx.in/_csp_xx.xx.xx.xx\",\"blocked-uri\":\"http://fonts.googleapis.com/css?family=Montserrat:400,700\",\"status-code\":0}}"}

This is my filbeat parser

filebeat.inputs.0:
- type: log
  enabled: true
  paths:
    - /var/log/csp.log
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
  json.keys_under_root: true
  json.add_error_key: true
processors:
  - decode_json_fields:
      fields: ["message"]

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

I even tried adjusting json pipeline

curl -k -u  'elastic:xxxxxx' -XPUT  "https://10.122.xx.xx:9200/_ingest/pipeline/json_pipeline" -H "Content-Type: application/json" -d '{
  "description": "Pipeline to parse nested request_body field",
  "processors": [
    {
      "json": {
        "field": "request_body",
        "target_field": "request_body",
        "add_to_root": false
      }
    }
  ]
}'

Alex_Salgado-Elastic · July 1, 2024, 10:25am

Have you checked the following points?

Blason · July 1, 2024, 11:41am

Yes - I read through those already but not sure if I am missing anything?

leandrojmp · July 1, 2024, 12:07pm

Check the first link, your configuration should be filebeat.inputs, not filebeat.inputs.0.

Have you tried using this? :

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp.log
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
  json.keys_under_root: true
  json.add_error_key: true

Also, is your source file a multiline file or does it have a json document per line?

Blason · July 1, 2024, 12:13pm

I spent so many days with that that

filebeat.inputs

And it did not work - while one of the google search pointed me like that and filebeat started clean.
And json is single line and not multiline - The multiline I kept it so that I was doing lot of R&D to get those stuff parsed

Blason · July 1, 2024, 12:16pm

OK here is my modified one

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp2.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: "message"
  #processors:
  # - decode_json_fields:
  #    fields: ["message"]

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

However still no luck

leandrojmp · July 1, 2024, 12:25pm

What do you have in filebeat logs? Also, share your entire filebeat.yml

You need to troubleshoot this step by step, first you need to make it arrive into Elasticsearch, then you start dealing with the parsing.

Also, where you want to parse it? In Filebeat or in Elasticsearch? Because you also have an ingest pipeline.

I would say that it would be easier to use filebeat just to send the logs and do all the parsing in Elasticsearch.

The csp2.log is a single line json?

What is the result of the command cat /var/log/csp2.log | wc -l on your server?

Blason · July 1, 2024, 12:42pm

cat /var/log/csp2.log | wc -l
1

And here is filebeat file

cat /etc/filebeat/filebeat.yml  | grep -v "#" | grep -vE '^
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp2.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: "message"
setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /tmp/ir.json
  json.keys_under_root: true
  json.add_error_key: true
  json.expand_keys: true
  json.overwrite_keys: true
processors:
  - decode_json_fields:
      fields: ["message"]
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["10.122.xxx.xx:9200"]
  ssl.certificate_authorities: ["/ca.crt"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"
  pipeline: "json_pipeline"
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Blason · July 1, 2024, 12:42pm

And ofcourse even I am looking to parse it in filebeat instead of elasticsearch. However since I am running out of option I am trying whatever I am aware of

leandrojmp · July 1, 2024, 12:56pm

Yeah, but to troubleshoot things you may need to take a step back to try to see what is happening and where is the error.

You didn't share the filebeat logs, please share the logs as well.

Also, your filebeat.yml has 3 filebeat.inputs, you should have just on filebeat.inputs and in then you configure each input (the type setting) as the example on the documentation.

I'm not sure how filebeat will work when you have multiple filebeat.inputs, I would assume that it would ignore the duplicates and use only the last one.

It seems that you have multiple lines duplicated, I'm also not sure how filebeat behaves in this situation.

You have 2 files in the configuration, ir.json and the csp2.log, are both of them not working?

I would take a step back and use this simplified filebeat.yml

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/csp2.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["10.122.xxx.xx:9200"]
  ssl.certificate_authorities: ["/ca.crt"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Then, after changing the filebeat.yml test both the config and the output.

Use filebeat test config and filebeat test output and share the results.

Blason · July 1, 2024, 1:02pm

elasticsearch: https://10.122.0.11:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.122.0.11
    dial up... OK
  TLS...
    security... WARN server's certificate chain verification is disabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.17.4

And here is

 filebeat test config
Config OK


2024-07-01T18:31:39.359+0530    INFO    instance/beat.go:685    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2024-07-01T18:31:39.359+0530    INFO    instance/beat.go:693    Beat ID: fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca
2024-07-01T18:31:39.362+0530    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2024-07-01T18:31:39.362+0530    INFO    [beat]  instance/beat.go:1039   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca"}}}
2024-07-01T18:31:39.362+0530    INFO    [beat]  instance/beat.go:1048   Build info      {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2024-07-01T18:31:39.362+0530    INFO    [beat]  instance/beat.go:1051   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.9"}}}
2024-07-01T18:31:39.363+0530    INFO    [beat]  instance/beat.go:1055   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-11-30T13:03:55+05:30","containerized":false,"name":"isn-siem","ip":["127.0.0.1/8","::1/128","139.59.15.218/20","10.47.0.14/16","fe80::a83c:ebff:fea9:d284/64","10.122.0.11/20","fe80::9816:98ff:fe04:3e7c/64"],"kernel_version":"5.4.0-167-generic","mac":["aa:3c:eb:a9:d2:84","9a:16:98:04:3e:7c"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"IST","timezone_offset_sec":19800,"id":"1a0aa414926b27af0d1e73d662a01ab4"}}}
2024-07-01T18:31:39.363+0530    INFO    [beat]  instance/beat.go:1084   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 2884214, "ppid": 2881357, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2024-07-01T18:31:39.200+0530"}}}
2024-07-01T18:31:39.363+0530    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.4
2024-07-01T18:31:39.363+0530    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.4' as ILM is enabled.
2024-07-01T18:31:39.364+0530    WARN    [cfgwarn]       tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2024-07-01T18:31:39.364+0530    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: https://10.122.0.11:9200
2024-07-01T18:31:39.364+0530    INFO    [publisher]     pipeline/module.go:113  Beat name: isn-siem
2024-07-01T18:31:39.366+0530    INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2024-07-01T18:31:39.366+0530    INFO    instance/beat.go:492    filebeat start running.
2024-07-01T18:31:39.367+0530    INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=43175470
2024-07-01T18:31:39.446+0530    INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=43186697
2024-07-01T18:31:39.447+0530    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 2
2024-07-01T18:31:39.447+0530    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 1
2024-07-01T18:31:39.447+0530    INFO    [crawler]       beater/crawler.go:117   starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.paths.0 filebeat.inputs.0.type]
2024-07-01T18:31:39.447+0530    ERROR   [input] input-logfile/manager.go:176    filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat
2024-07-01T18:31:39.447+0530    INFO    [crawler]       beater/crawler.go:148   Starting input (ID: 17228493714276037636)
2024-07-01T18:31:39.447+0530    INFO    [input.filestream]      compat/compat.go:111    Input filestream starting       {"id": "EF17E7C0AE793404"}
2024-07-01T18:31:39.448+0530    INFO    [file_watcher]  filestream/fswatch.go:140       Start next scan
2024-07-01T18:31:39.454+0530    INFO    [crawler]       beater/crawler.go:106   Loading and starting Inputs completed. Enabled inputs: 1
2024-07-01T18:31:39.454+0530    INFO    cfgfile/reload.go:164   Config reloader started
2024-07-01T18:31:39.464+0530    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: https://10.122.0.11:9200
2024-07-01T18:31:39.470+0530    INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2024-07-01T18:31:39.475+0530    INFO    cfgfile/reload.go:224   Loading of config files completed.
2024-07-01T18:31:39.475+0530    INFO    [input.httpjson-cursor] compat/compat.go:111    Input httpjson-cursor starting  {"id": "F28AEA1431B7F788"}
2024-07-01T18:31:39.475+0530    INFO    [input.httpjson-stateless]      compat/compat.go:111    Input httpjson-stateless starting       {"id": "A3B2CB572CE313E5"}
2024-07-01T18:31:39.475+0530    INFO    [input.httpjson-stateless]      compat/compat.go:111    Input httpjson-stateless starting       {"id": "86F57B6043A0C821"}
2024-07-01T18:31:39.475+0530    INFO    [input.httpjson-stateless]      compat/compat.go:111    Input httpjson-stateless starting       {"id": "F8929C8C8CD1AAC0"}
2024-07-01T18:31:39.475+0530    INFO    [input.httpjson-stateless]      v2/input.go:112 Process another repeated request.       {"id": "A3B2CB572CE313E5", "input_url": "https://urlhaus-api.abuse.ch/v1/urls/recent/"}
2024-07-01T18:31:39.475+0530    INFO    [input.httpjson-stateless]      v2/input.go:112 Process another repeated request.       {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2024-07-01T18:31:39.476+0530    INFO    [input.httpjson-stateless]      v2/input.go:112 Process another repeated request.       {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2024-07-01T18:31:39.476+0530    INFO    [input.httpjson-cursor] v2/input.go:112 Process another repeated request.       {"id": "F28AEA1431B7F788", "input_source": "https://otx.alienvault.com/api/v1/indicators/export", "input_url": "https://otx.alienvault.com/api/v1/indicators/export"}
2024-07-01T18:31:40.448+0530    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://10.122.0.11:9200))
2024-07-01T18:31:40.449+0530    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-07-01T18:31:40.449+0530    INFO    [publisher]     pipeline/retry.go:223     done
2024-07-01T18:31:40.454+0530    INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2024-07-01T18:31:40.471+0530    INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2024-07-01T18:31:40.485+0530    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-07-01T18:31:40.492+0530    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy filebeat exists already.
2024-07-01T18:31:40.492+0530    INFO    [index-management]      idxmgmt/std.go:397      Set setup.template.name to '{filebeat-7.17.4 {now/d}-000001}' as ILM is enabled.
2024-07-01T18:31:40.492+0530    INFO    [index-management]      idxmgmt/std.go:402      Set setup.template.pattern to 'filebeat-7.17.4-*' as ILM is enabled.
2024-07-01T18:31:40.492+0530    INFO    [index-management]      idxmgmt/std.go:436      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.4 {now/d}-000001} as ILM is enabled.
2024-07-01T18:31:40.492+0530    INFO    [index-management]      idxmgmt/std.go:440      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2024-07-01T18:31:40.509+0530    INFO    template/load.go:110    Template "filebeat-7.17.4" already exists and will not be overwritten.
2024-07-01T18:31:40.509+0530    INFO    [index-management]      idxmgmt/std.go:297      Loaded index template.
2024-07-01T18:31:40.512+0530    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias filebeat-7.17.4 exists already.
2024-07-01T18:31:40.519+0530    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://10.122.0.11:9200)) established
2024-07-01T18:31:40.935+0530    INFO    [input.httpjson-stateless]      v2/request.go:204       request finished: 1000 events published {"id": "A3B2CB572CE313E5", "input_url": "https://urlhaus-api.abuse.ch/v1/urls/recent/"}
2024-07-01T18:31:42.167+0530    INFO    [input.httpjson-stateless]      v2/request.go:204       request finished: 48 events published   {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2024-07-01T18:31:42.816+0530    WARN    [elasticsearch] elasticsearch/client.go:414     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.July, 1, 18, 31, 42, 164873675, time.Local), Meta:{"_id":"a9babe52240c6472986cc2e8f58ef8aa37cabe307fac15101569cc5ab953e8ef","pipeline":"filebeat-7.17.4-threatintel-malwarebazaar-pipeline"}, Fields:{"agent":{"ephemeral_id":"52d736a5-5ed2-40f5-b709-742488c324f4","hostname":"isn-siem","id":"fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca","name":"isn-siem","type":"filebeat","version":"7.17.4"},"ecs":{"version":"1.12.0"},"event":{"created":"2024-07-01T13:01:42.164Z","dataset":"threatintel.malwarebazaar","module":"threatintel"},"fileset":{"name":"malwarebazaar"},"input":{"type":"httpjson"},"json":{"anonymous":0,"code_sign":[{"algorithm":"sha256WithRSAEncryption","issuer_cn":"Fjeldskreddets","serial_number":"55432c16faecc698c83e18bd4edfa7b279218f7b","subject_cn":"Fjeldskreddets","thumbprint":"a98702daf3f959c9f17852d76641ee02cfab306351b8b8539c98780b2cc21565","thumbprint_algorithm":"SHA256","valid_from":"2023-10-17T07:04:51Z","valid_to":"2026-10-16T07:04:51Z"}],"dhash_icon":"a5a026f1e0f2f230","file_name":"SeAH RFP_24-0676·pdf.exe","file_size":486976,"file_type":"exe","file_type_mime":"application/x-dosexec","first_seen":"2024-07-01 12:50:01","gimphash":null,"imphash":"b40f29cd171eb54c01b1dd2683c9c26b","intelligence":{"clamav":null,"downloads":"232","mail":null,"uploads":"1"},"last_seen":null,"md5_hash":"8951c491b26675b308464af7a29567bd","origin_country":"HU","reporter":"adrian__luca","sha1_hash":"883bd6d014e3baf9141b304519fc34eef20eb41f","sha256_hash":"3fc5e4d002e04269f2f674e6a2e98935df133ffe0f1fd54c817662d864c2f1b8","sha3_384_hash":"edf5e838b61c9948da7ea4a348eba43ac482625013c8053f6a7126b2143a08e4279b89afaab73f69c927f92b6dca70e6","signature":"Loki","ssdeep":"12288:/qgowhL+Pylw1QeQMQukEsvs5uT8JGjD8WpyLk8n:xR+a3M0ee8JG38lr","tags":["exe","Loki","signed"],"telfhash":null,"tlsh":"T1FAA4F187F6049076E51DACF34B6FC66FA92B6F40363A0603D7D07A2A067D6F66B13046"},"message":"{\"anonymous\":0,\"code_sign\":[{\"algorithm\":\"sha256WithRSAEncryption\",\"issuer_cn\":\"Fjeldskreddets\",\"serial_number\":\"55432c16faecc698c83e18bd4edfa7b279218f7b\",\"subject_cn\":\"Fjeldskreddets\",\"thumbprint\":\"a98702daf3f959c9f17852d76641ee02cfab306351b8b8539c98780b2cc21565\",\"thumbprint_algorithm\":\"SHA256\",\"valid_from\":\"2023-10-17T07:04:51Z\",\"valid_to\":\"2026-10-16T07:04:51Z\"}],\"dhash_icon\":\"a5a026f1e0f2f230\",\"file_name\":\"SeAH RFP_24-0676·pdf.exe\",\"file_size\":486976,\"file_type\":\"exe\",\"file_type_mime\":\"application/x-dosexec\",\"first_seen\":\"2024-07-01 12:50:01\",\"gimphash\":null,\"imphash\":\"b40f29cd171eb54c01b1dd2683c9c26b\",\"intelligence\":{\"clamav\":null,\"downloads\":\"232\",\"mail\":null,\"uploads\":\"1\"},\"last_seen\":null,\"md5_hash\":\"8951c491b26675b308464af7a29567bd\",\"origin_country\":\"HU\",\"reporter\":\"adrian__luca\",\"sha1_hash\":\"883bd6d014e3baf9141b304519fc34eef20eb41f\",\"sha256_hash\":\"3fc5e4d002e04269f2f674e6a2e98935df133ffe0f1fd54c817662d864c2f1b8\",\"sha3_384_hash\":\"edf5e838b61c9948da7ea4a348eba43ac482625013c8053f6a7126b2143a08e4279b89afaab73f69c927f92b6dca70e6\",\"signature\":\"Loki\",\"ssdeep\":\"12288:/qgowhL+Pylw1QeQMQukEsvs5uT8JGjD8WpyLk8n:xR+a3M0ee8JG38lr\",\"tags\":[\"exe\",\"Loki\",\"signed\"],\"telfhash\":null,\"tlsh\":\"T1FAA4F187F6049076E51DACF34B6FC66FA92B6F40363A0603D7D07A2A067D6F66B13046\"}","service":{"type":"threatintel"},"tags":["threatintel-malwarebazaar","forwarded"]}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [threatintel.malwarebazaar.code_sign] of type [keyword] in document with id 'a9babe52240c6472986cc2e8f58ef8aa37cabe307fac15101569cc5ab953e8ef'. Preview of field's value: '{subject_cn=Fjeldskreddets, issuer_cn=Fjeldskreddets, valid_to=2026-10-16T07:04:51Z, thumbprint=a98702daf3f959c9f17852d76641ee02cfab306351b8b8539c98780b2cc21565, valid_from=2023-10-17T07:04:51Z, serial_number=55432c16faecc698c83e18bd4edfa7b279218f7b, thumbprint_algorithm=SHA256, algorithm=sha256WithRSAEncryption}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:2511"}}, dropping event!

Blason · July 1, 2024, 1:06pm

After this - Json messages does not get parsed and all those messages gets in message field

{
  "_index": "filebeat-7.17.4-2024.06.27-000105",
  "_type": "_doc",
  "_id": "McxjbpAB0yAvttXtAX0o",
  "_version": 1,
  "_score": 1,
  "_source": {
    "@timestamp": "2024-07-01T13:01:39.448Z",
    "ecs": {
      "version": "1.12.0"
    },
    "host": {
      "id": "1a0aa414926b27af0d1e73d662a01ab4",
      "containerized": false,
      "ip": [
        "10.47.0.14",
        "fe80::a83c:ebff:fea9:d284",
        "10.122.0.11",
        "fe80::9816:98ff:fe04:3e7c"
      ],
      "mac": [
        "aa:3c:eb:a9:d2:84",
        "9a:16:98:04:3e:7c"
      ],
      "hostname": "isn-siem",
      "architecture": "x86_64",
      "name": "isn-siem",
      "os": {
        "name": "Ubuntu",
        "kernel": "5.4.0-167-generic",
        "codename": "focal",
        "type": "linux",
        "platform": "ubuntu",
        "version": "20.04.4 LTS (Focal Fossa)",
        "family": "debian"
      }
    },
    "agent": {
      "hostname": "isn-siem",
      "ephemeral_id": "52d736a5-5ed2-40f5-b709-742488c324f4",
      "id": "fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca",
      "name": "isn-siem",
      "type": "filebeat",
      "version": "7.17.4"
    },
    "log": {
      "file": {
        "path": "/var/log/csp2.log"
      },
      "offset": 0
    },
    "message": "{\"date\":\"01/Jul/2024:11:50:43 +0530\",\"IP address\":\"49.xx.xx.xx\",\"http_x_forwarded_for\":\"\",\"status\":\"204\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15\",\"body_bytes_sent\":\"0\",\"request\":\"POST /_csp_xx.xx.xx.xx HTTP/1.1\",\"request_body\":\"{\\\"type\\\":\\\"csp-violation\\\",\\\"url\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"csp-report\\\":{\\\"document-uri\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"referrer\\\":\\\"https://www.google.com/\\\",\\\"violated-directive\\\":\\\"style-src-elem\\\",\\\"effective-directive\\\":\\\"style-src-elem\\\",\\\"original-policy\\\":\\\"default-src 'self' 'unsafe-inline'; frame-src 'self' td.doubleclick.net;  manifest-src 'self' www.xx.xx.xx.xx.com;  style-src 'self' 'unsafe-inline' *.googleapis.com 'self' fonts.googleapis.com;  script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' https://www.google-analytics.com 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-C8oQVJ33cKtnkARnmeWp6SDChkU+u7KvsNMFUzkkUzk=' 'sha256-TXAWuIqKdrNpSKYHYZ7bkGoIMlHGjhBceAOn7h5QUQw=' 'sha256-tqlMHUh+wHh08rh4nIZeMbwnBAcss9QVE8OKfdrvodU=' 'sha256-2b5RU9WZsUgm7tNV36A0w17RAyEyqDvuxTzvHVIQ6E0=' 'sha256-NZe6EI6DHYFUR8E7IB0jYeyXL+6P2HQzsVridcWVESE=' 'sha256-9l867tRreZQISfogIvLL3zaCiN6QRvKz2gdcQvnd6PY=' 'sha256-FXSlRA54YYcbC4EqhWvLY4A55v1v4ONi1Rk2lHMJZf8=' 'sha256-8ijKzAGJMbNBQIOqLZ3pM/92KPYjh08Mm/QzYPSJ2e8=' 'sha256-WMOEx/fVLpdP2x5+htiQ6TwHSKlY31r0FFIbtbfIfIM=' ajax.googleapis.com https://www.xx.xx.xx.xx.com;  script-src-attr 'self';  style-src-attr 'self' 'unsafe-inline';  script-src-elem 'self' 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' https://www.googletagmanager.com http://fonts.googleapis.com https://www.google-analytics.com https://ajax.googleapis.com;  style-src-elem 'self' 'unsafe-inline' www.xx.xx.xx.xx.com gc.kis.v2.scr.kaspersky-labs.comi http://fonts.googleapis.com;  media-src 'self' data:;  connect-src 'self' analytics.google.com stats.g.doubleclick.net www.google-analytics.com www.google.com.sg www.google.co.in region1.analytics.google.com region1.google-analytics.com www.google.co.jp www.google.co.th www.google.nl www.google.fr www.google.com.ng www.google.ae www.google.com.hk;  font-src 'self' www.xx.xx.xx.xx.com http://fonts.gstatic.com http://fonts.gstatic.com data:;  img-src 'self' www.xx.xx.xx.xx.com www.google.co.in www.google.com www.google-analytics.com www.google.co.uk www.googletagmanager.com www.google.co.jp www.google.ru www.google.co.th www.google.com.om analytics.google.com stats.g.doubleclick.net www.google.co.kr www.google.cz www.google.nl www.google.fr www.google.com.sa www.google.com.bh data: www.google.com.mx www.google.com.au www.google.com.ng www.google.com.sg www.google.ae www.google.de www.google.com.hk; object-src 'none'; base-uri 'self'; upgrade-insecure-requests; report-uri https://csp.xx.xx.xx.xx/_csp_xx.xx.xx.xx\\\",\\\"blocked-uri\\\":\\\"http://fonts.googleapis.com/css?family=Open+Sans:400,300\\\",\\\"status-code\\\":0}}\"}",
    "input": {
      "type": "filestream"
    }
  },
  "fields": {
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "is-siem"
    ],
    "host.mac": [
      "aa:3c:eb:a9:d2:84",
      "9a:16:98:04:3e:7c"
    ],
    "host.ip": [
      ,
      "10.47.0.14",
      "fe80::a83c:ebff:fea9:d284",
      "10.122.xx.xx",
      "fe80::9816:98ff:fe04:3e7c"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.version": [
      "20.04.4 LTS (Focal Fossa)"
    ],
    "host.os.kernel": [
      "5.4.0-167-generic"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "isn-siem"
    ],
    "host.name": [
      "isn-siem"
    ],
    "host.id": [
      "1a0aa414926b27af0d1e73d662a01ab4"
    ],
    "host.os.type": [
      "linux"
    ],
    "host.os.codename": [
      "focal"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      0
    ],
    "agent.hostname": [
      "isn-siem"
    ],
    "message": [
      "{\"date\":\"01/Jul/2024:11:50:43 +0530\",\"IP address\":\"49.37.8.152\",\"http_x_forwarded_for\":\"\",\"status\":\"204\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15\",\"body_bytes_sent\":\"0\",\"request\":\"POST /_csp_xx.xx.xx.xx HTTP/1.1\",\"request_body\":\"{\\\"type\\\":\\\"csp-violation\\\",\\\"url\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"csp-report\\\":{\\\"document-uri\\\":\\\"https://www.xx.xx.xx.xx.com/\\\",\\\"referrer\\\":\\\"https://www.google.com/\\\",\\\"violated-directive\\\":\\\"style-src-elem\\\",\\\"effective-directive\\\":\\\"style-src-elem\\\",\\\"original-policy\\\":\\\"default-src 'self' 'unsafe-inline'; frame-src 'self' td.doubleclick.net;  manifest-src 'self' www.xx.xx.xx.xx.com;  style-src 'self' 'unsafe-inline' *.googleapis.com 'self' fonts.googleapis.com;  script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' https://www.google-analytics.com 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-C8oQVJ33cKtnkARnmeWp6SDChkU+u7KvsNMFUzkkUzk=' 'sha256-TXAWuIqKdrNpSKYHYZ7bkGoIMlHGjhBceAOn7h5QUQw=' 'sha256-tqlMHUh+wHh08rh4nIZeMbwnBAcss9QVE8OKfdrvodU=' 'sha256-2b5RU9WZsUgm7tNV36A0w17RAyEyqDvuxTzvHVIQ6E0=' 'sha256-NZe6EI6DHYFUR8E7IB0jYeyXL+6P2HQzsVridcWVESE=' 'sha256-9l867tRreZQISfogIvLL3zaCiN6QRvKz2gdcQvnd6PY=' 'sha256-FXSlRA54YYcbC4EqhWvLY4A55v1v4ONi1Rk2lHMJZf8=' 'sha256-8ijKzAGJMbNBQIOqLZ3pM/92KPYjh08Mm/QzYPSJ2e8=' 'sha256-WMOEx/fVLpdP2x5+htiQ6TwHSKlY31r0FFIbtbfIfIM=' ajax.googleapis.com https://www.xx.xx.xx.xx.com;  script-src-attr 'self';  style-src-attr 'self' 'unsafe-inline';  script-src-elem 'self' 'sha256-chKR6/W5cKbDvAyQg+E5NqGqVRJMsneBQNnAvDnBAlQ=' 'sha256-3jblAZQyCn0+8azpvTSodai9RYslPAYZed1ijpv0mv0=' 'sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE=' 'sha256-4Z6RsPSZvVIZt/SwrbTqoZjubKsnn9KVPwIVnQjyNJw=' 'sha256-Nh7fO9U0UwzFwDrPHSBdcI9YVErvPzpo0q3DDJE0M1w=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' 'sha256-uqx66C0m1v7mX6JkNTa/EfRziNpNW6jZJj6b8csfYKg=' 'sha256-kHv8ZHTpjrzvmzY1qFRqhFC4LBuI9INyfyGDGk3VwWw=' https://www.googletagmanager.com http://fonts.googleapis.com https://www.google-analytics.com https://ajax.googleapis.com;  style-src-elem 'self' 'unsafe-inline' www.xx.xx.xx.xx.com gc.kis.v2.scr.kaspersky-labs.comi http://fonts.googleapis.com;  media-src 'self' data:;  connect-src 'self' analytics.google.com stats.g.doubleclick.net www.google-analytics.com www.google.com.sg www.google.co.in region1.analytics.google.com region1.google-analytics.com www.google.co.jp www.google.co.th www.google.nl www.google.fr www.google.com.ng www.google.ae www.google.com.hk;  font-src 'self' www.xx.xx.xx.xx.com http://fonts.gstatic.com http://fonts.gstatic.com data:;  img-src 'self' www.xx.xx.xx.xx.com www.google.co.in www.google.com www.google-analytics.com www.google.co.uk www.googletagmanager.com www.google.co.jp www.google.ru www.google.co.th www.google.com.om analytics.google.com stats.g.doubleclick.net www.google.co.kr www.google.cz www.google.nl www.google.fr www.google.com.sa www.google.com.bh data: www.google.com.mx www.google.com.au www.google.com.ng www.google.com.sg www.google.ae www.google.de www.google.com.hk; object-src 'none'; base-uri 'self'; upgrade-insecure-requests; report-uri https://csp.xx.xx.xx.xx/_csp_xx.xx.xx.xx\\\",\\\"blocked-uri\\\":\\\"http://fonts.googleapis.com/css?family=Open+Sans:400,300\\\",\\\"status-code\\\":0}}\"}"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "@timestamp": [
      "2024-07-01T13:01:39.448Z"
    ],
    "agent.id": [
      "fbfb1d8e-7ada-4ab8-bb0f-c692c7702fca"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "host.containerized": [
      false
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.file.path": [
      "/var/log/csp2.log"
    ],
    "agent.ephemeral_id": [
      "52d736a5-5ed2-40f5-b709-742488c324f4"
    ],
    "agent.version": [
      "7.17.4"
    ],
    "host.os.family": [
      "debian"
    ]
  }
}

Blason · July 2, 2024, 6:29am

Still no luck

Any ideas folks - can someone please help me in this configuration?

leandrojmp · July 2, 2024, 1:49pm

What is the filebeat.yml that you are using now? The one I shared?

Since your messages are arriving at Elasticsearch, you now can try adding the processor in filebeat to parse it.

Try adding this to your filebeat.yml

processors:
  - decode_json_fields:
      fields: "message"
      max_depth: 2
      target: "parsed_json"
      overwrite_keys: false
      add_error_key: true

This would parse your message field, up to 2 nested levels, and store the result in a field named parsed_json.

Blason · July 3, 2024, 2:55am

Yes I am using your json and let me try adding that field.

Blason · July 3, 2024, 3:12am

Now this is my error

And here is my file

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/csp2.log
      #json.keys_under_root: true
  json.add_error_key: true
  #  json.message_key: "request_body"
processors:
  - decode_json_fields:
      fields: "message"
      max_depth: 2
      target: "parsed_json"
      overwrite_keys: false
      add_error_key: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["10.122.0.11:9200"]
  ssl.certificate_authorities: ["/ca.crt"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~