Anomali threat intel dashboard not populating

Charles_Allen · June 6, 2022, 8:19pm

Hello, I am trying to see Anomali threat intel data in kibana but the dashboard is just empty. I am running Elasticsearch 7.17.4

I have uncommented the anomali guest account and password.

I am seeing results within the malwarebazzar threat intel dashboard so it doesn't seem to be an issue with filebeat itself.

Here is a copy of the anomali section of my threatintel.yml

  anomali:
    enabled: true

    # Input used for ingesting threat intel data
    var.input: httpjson

    # The URL used for Threat Intel API calls. Limo has multiple different possibilities for URL's depending
    # on the type of threat intel source that is needed.
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects

    # The Username used by anomali Limo, defaults to guest.
    var.username: guest

    # The password used by anomali Limo, defaults to guest.
    var.password: guest

    # How far back to look once the beat starts up for the first time, the value has to be in hours.
    var.first_interval: 400h

    # The interval to poll the API for updates
    var.interval: 5m

I am new to elastic so let me know if I can provide anymore information

legoguy1000 · June 7, 2022, 2:16am

Can u provide the logs from filebeat

Charles_Allen · June 7, 2022, 2:39am

[root@elastic filebeat]# filebeat -e
2022-06-07T02:35:19.797Z        INFO    instance/beat.go:685    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-06-07T02:35:19.797Z        INFO    instance/beat.go:693    Beat ID: 304c5e9c-7bda-4308-aaf2-bf8e67b93eaf
2022-06-07T02:35:19.799Z        WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: network is unreachable. No token in the metadata request will be used.
2022-06-07T02:35:19.800Z        INFO    instance/beat.go:425    filebeat stopped.
2022-06-07T02:35:19.800Z        ERROR   instance/beat.go:1014   Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
[root@elastic filebeat]# systemctl stop filebeat
[root@elastic filebeat]# filebeat -e
2022-06-07T02:35:30.248Z        INFO    instance/beat.go:685    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-06-07T02:35:30.248Z        INFO    instance/beat.go:693    Beat ID: 304c5e9c-7bda-4308-aaf2-bf8e67b93eaf
2022-06-07T02:35:30.251Z        WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: network is unreachable. No token in the metadata request will be used.
2022-06-07T02:35:30.252Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2022-06-07T02:35:30.252Z        INFO    [beat]  instance/beat.go:1039   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "304c5e9c-7bda-4308-aaf2-bf8e67b93eaf"}}}
2022-06-07T02:35:30.252Z        INFO    [beat]  instance/beat.go:1048   Build info      {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2022-06-07T02:35:30.252Z        INFO    [beat]  instance/beat.go:1051   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.17.9"}}}
2022-06-07T02:35:30.252Z        INFO    [beat]  instance/beat.go:1055   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-06-06T19:40:40Z","containerized":false,"name":"elastic.local","ip":["127.0.0.1/8","::1/128","10.0.0.101/24","fe80::a00:27ff:fe77:d8db/64","10.0.3.15/24","fe80::a00:27ff:fe77:a598/64"],"kernel_version":"4.18.0-383.el8.x86_64","mac":["08:00:27:77:d8:db","08:00:27:77:a5:98"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Stream","version":"8","major":8,"minor":0,"patch":0},"timezone":"UTC","timezone_offset_sec":0,"id":"a80ccfe94c474a9eb0c565871ef0ca0c"}}}
2022-06-07T02:35:30.252Z        INFO    [beat]  instance/beat.go:1084   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 4237, "ppid": 1682, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-06-07T02:35:30.080Z"}}}
2022-06-07T02:35:30.253Z        INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.4
2022-06-07T02:35:30.253Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.4' as ILM is enabled.
2022-06-07T02:35:30.253Z        INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: http://localhost:9200
2022-06-07T02:35:30.254Z        INFO    [publisher]     pipeline/module.go:113  Beat name: elastic.local
2022-06-07T02:35:30.254Z        INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2022-06-07T02:35:30.254Z        INFO    instance/beat.go:492    filebeat start running.
2022-06-07T02:35:30.255Z        INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2022-06-07T02:35:30.255Z        INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=11
2022-06-07T02:35:30.255Z        INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2022-06-07T02:35:30.255Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 1
2022-06-07T02:35:30.255Z        INFO    [crawler]       beater/crawler.go:117   starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]
2022-06-07T02:35:30.255Z        INFO    [crawler]       beater/crawler.go:121   input disabled, skipping it
2022-06-07T02:35:30.259Z        WARN    [input] v2/loader.go:104        BETA: The http_endpoint input is beta   {"input": "http_endpoint", "stability": "Beta", "deprecated": false}
2022-06-07T02:35:30.260Z        INFO    [crawler]       beater/crawler.go:106   Loading and starting Inputs completed. Enabled inputs: 0
2022-06-07T02:35:30.260Z        INFO    cfgfile/reload.go:164   Config reloader started
2022-06-07T02:35:30.265Z        WARN    [input] v2/loader.go:104        BETA: The http_endpoint input is beta   {"input": "http_endpoint", "stability": "Beta", "deprecated": false}
2022-06-07T02:35:30.266Z        INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: http://localhost:9200
2022-06-07T02:35:30.267Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2022-06-07T02:35:30.269Z        INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2022-06-07T02:35:30.277Z        INFO    cfgfile/reload.go:224   Loading of config files completed.
2022-06-07T02:35:30.277Z        INFO    [input.httpjson-cursor] compat/compat.go:111    Input httpjson-cursor starting  {"id": "5B4CB7C8F10729D1"}
2022-06-07T02:35:30.277Z        INFO    [input.httpjson-cursor] v2/input.go:112 Process another repeated request.       {"id": "5B4CB7C8F10729D1", "input_source": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc", "input_url": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc"}
2022-06-07T02:35:30.278Z        INFO    [input.httpjson-stateless]      compat/compat.go:111    Input httpjson-stateless starting       {"id": "F8929C8C8CD1AAC0"}
2022-06-07T02:35:30.278Z        INFO    [input.httpjson-cursor] compat/compat.go:111    Input httpjson-cursor starting  {"id": "99629170FE3D3B02"}
2022-06-07T02:35:30.278Z        INFO    [input.http_endpoint]   compat/compat.go:111    Input http_endpoint starting    {"id": "885D95D6F64FD52C"}
2022-06-07T02:35:30.278Z        INFO    [input.httpjson-stateless]      compat/compat.go:111    Input httpjson-stateless starting       {"id": "86F57B6043A0C821"}
2022-06-07T02:35:30.278Z        INFO    [input.httpjson-stateless]      v2/input.go:112 Process another repeated request.       {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2022-06-07T02:35:30.278Z        INFO    [input.httpjson-stateless]      v2/input.go:112 Process another repeated request.       {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:30.279Z        INFO    [input.httpjson-cursor] v2/input.go:112 Process another repeated request.       {"id": "99629170FE3D3B02", "input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects"}
2022-06-07T02:35:30.279Z        INFO    [input.http_endpoint]   http_endpoint/input.go:121      Starting HTTP server on localhost:8080  {"id": "885D95D6F64FD52C", "address": "localhost:8080"}
2022-06-07T02:35:30.509Z        ERROR   [input.httpjson-stateless.retryablehttp]        go-retryablehttp@v0.6.6/client.go:553   request failed%!(EXTRA string=error, *url.Error=Get "https://urlhaus-api.abuse.ch/v1/payloads/recent/": tls: first record does not look like a TLS handshake, string=method, string=GET, string=url, *url.URL=https://urlhaus-api.abuse.ch/v1/payloads/recent/)  {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:30.583Z        ERROR   [input.httpjson-cursor] v2/input.go:115 Error while processing http request: failed to execute http client.Do: server responded with status code 401: {"error":{"status":401}}                                                                                          {"id": "5B4CB7C8F10729D1", "input_source": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc", "input_url": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc"}
2022-06-07T02:35:30.955Z        INFO    [input.httpjson-stateless]      v2/request.go:204       request finished: 9 events published    {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2022-06-07T02:35:31.713Z        ERROR   [input.httpjson-stateless.retryablehttp]        go-retryablehttp@v0.6.6/client.go:553   request failed%!(EXTRA string=error, *url.Error=Get "https://urlhaus-api.abuse.ch/v1/payloads/recent/": tls: first record does not look like a TLS handshake, string=method, string=GET, string=url, *url.URL=https://urlhaus-api.abuse.ch/v1/payloads/recent/)  {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:31.955Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(http://localhost:9200))
2022-06-07T02:35:31.956Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-06-07T02:35:31.956Z        INFO    [publisher]     pipeline/retry.go:223     done
2022-06-07T02:35:31.957Z        INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2022-06-07T02:35:31.975Z        INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2022-06-07T02:35:31.992Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2022-06-07T02:35:32.000Z        INFO    [index-management.ilm]  ilm/std.go:170  ILM policy filebeat exists already.
2022-06-07T02:35:32.000Z        INFO    [index-management]      idxmgmt/std.go:397      Set setup.template.name to '{filebeat-7.17.4 {now/d}-000001}' as ILM is enabled.
2022-06-07T02:35:32.000Z        INFO    [index-management]      idxmgmt/std.go:402      Set setup.template.pattern to 'filebeat-7.17.4-*' as ILM is enabled.
2022-06-07T02:35:32.000Z        INFO    [index-management]      idxmgmt/std.go:436      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.4 {now/d}-000001} as ILM is enabled.
2022-06-07T02:35:32.000Z        INFO    [index-management]      idxmgmt/std.go:440      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-06-07T02:35:32.013Z        INFO    template/load.go:110    Template "filebeat-7.17.4" already exists and will not be overwritten.
2022-06-07T02:35:32.013Z        INFO    [index-management]      idxmgmt/std.go:297      Loaded index template.
2022-06-07T02:35:32.015Z        INFO    [index-management.ilm]  ilm/std.go:126  Index Alias filebeat-7.17.4 exists already.
2022-06-07T02:35:32.025Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(http://localhost:9200)) established
2022-06-07T02:35:32.445Z        INFO    [input.httpjson-cursor] v2/request.go:204       request finished: 0 events published    {"id": "99629170FE3D3B02", "input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects"}
2022-06-07T02:35:33.872Z        ERROR   [input.httpjson-stateless.retryablehttp]        go-retryablehttp@v0.6.6/client.go:553   request failed%!(EXTRA string=error, *url.Error=Get "https://urlhaus-api.abuse.ch/v1/payloads/recent/": tls: first record does not look like a TLS handshake, string=method, string=GET, string=url, *url.URL=https://urlhaus-api.abuse.ch/v1/payloads/recent/)  {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:38.074Z        ERROR   [input.httpjson-stateless.retryablehttp]        go-retryablehttp@v0.6.6/client.go:553   request failed%!(EXTRA string=error, *url.Error=Get "https://urlhaus-api.abuse.ch/v1/payloads/recent/": tls: first record does not look like a TLS handshake, string=method, string=GET, string=url, *url.URL=https://urlhaus-api.abuse.ch/v1/payloads/recent/)  {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:46.245Z        ERROR   [input.httpjson-stateless.retryablehttp]        go-retryablehttp@v0.6.6/client.go:553   request failed%!(EXTRA string=error, *url.Error=Get "https://urlhaus-api.abuse.ch/v1/payloads/recent/": tls: first record does not look like a TLS handshake, string=method, string=GET, string=url, *url.URL=https://urlhaus-api.abuse.ch/v1/payloads/recent/)  {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
q
^C2022-06-07T02:35:57.143Z      INFO    beater/filebeat.go:553  Stopping filebeat
2022-06-07T02:35:57.143Z        INFO    beater/crawler.go:155   Stopping Crawler
2022-06-07T02:35:57.143Z        INFO    beater/crawler.go:165   Stopping 0 inputs
2022-06-07T02:35:57.143Z        INFO    cfgfile/reload.go:227   Dynamic config reloader stopped
2022-06-07T02:35:57.143Z        INFO    [reload]        cfgfile/list.go:129     Stopping 1 runners ...
2022-06-07T02:35:57.143Z        INFO    [input.httpjson-stateless]      compat/compat.go:132    Input 'httpjson-stateless' stopped      {"id": "F8929C8C8CD1AAC0"}
2022-06-07T02:35:57.143Z        INFO    [input.httpjson-cursor] compat/compat.go:132    Input 'httpjson-cursor' stopped {"id": "99629170FE3D3B02"}
2022-06-07T02:35:57.143Z        INFO    [input.http_endpoint]   compat/compat.go:132    Input 'http_endpoint' stopped   {"id": "885D95D6F64FD52C"}
2022-06-07T02:35:57.143Z        INFO    [input.httpjson-stateless]      compat/compat.go:132    Input 'httpjson-stateless' stopped      {"id": "86F57B6043A0C821"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-cursor] compat/compat.go:132    Input 'httpjson-cursor' stopped {"id": "5B4CB7C8F10729D1"}
2022-06-07T02:35:57.144Z        INFO    beater/crawler.go:185   Crawler stopped
2022-06-07T02:35:57.144Z        INFO    [registrar]     registrar/registrar.go:132      Stopping Registrar
2022-06-07T02:35:57.144Z        INFO    [registrar]     registrar/registrar.go:166      Ending Registrar
2022-06-07T02:35:57.144Z        INFO    [registrar]     registrar/registrar.go:137      Registrar stopped
2022-06-07T02:35:57.144Z        ERROR   [input.httpjson-stateless]      v2/input.go:115 Error while processing http request: failed to execute http client.Do: failed to execute http client.Do: Get "https://urlhaus-api.abuse.ch/v1/payloads/recent/": context canceled                                       {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-stateless]      v2/input.go:131 Input stopped because context was cancelled with: context canceled                                                                                                                                                      {"id": "F8929C8C8CD1AAC0", "input_url": "https://urlhaus-api.abuse.ch/v1/payloads/recent/"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-stateless]      compat/compat.go:124    Input 'httpjson-stateless' stopped      {"id": "F8929C8C8CD1AAC0"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-cursor] v2/input.go:131 Input stopped because context was cancelled with: context canceled                                                                                                                                                              {"id": "99629170FE3D3B02", "input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-cursor] compat/compat.go:124    Input 'httpjson-cursor' stopped {"id": "99629170FE3D3B02"}
2022-06-07T02:35:57.144Z        INFO    [input.http_endpoint]   compat/compat.go:124    Input 'http_endpoint' stopped   {"id": "885D95D6F64FD52C"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-stateless]      v2/input.go:131 Input stopped because context was cancelled with: context canceled                                                                                                                                                      {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-stateless]      compat/compat.go:124    Input 'httpjson-stateless' stopped      {"id": "86F57B6043A0C821"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-cursor] v2/input.go:131 Input stopped because context was cancelled with: context canceled                                                                                                                                                              {"id": "5B4CB7C8F10729D1", "input_source": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc", "input_url": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc"}
2022-06-07T02:35:57.144Z        INFO    [input.httpjson-cursor] compat/compat.go:124    Input 'httpjson-cursor' stopped {"id": "5B4CB7C8F10729D1"}
2022-06-07T02:35:57.151Z        INFO    [monitoring]    log/log.go:192  Total metrics   {"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"id":"session-1.scope","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":68587520}}}},"cpu":{"system":{"ticks":20,"time":{"ms":29}},"total":{"ticks":140,"time":{"ms":157},"value":140},"user":{"ticks":120,"time":{"ms":128}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":13},"info":{"ephemeral_id":"b1ed9505-b148-4fb8-843f-922d6c847802","uptime":{"ms":26951},"version":"7.17.4"},"memstats":{"gc_next":26170656,"memory_alloc":16945296,"memory_sys":33375240,"memory_total":68538224,"rss":137134080},"runtime":{"goroutines":18}},"filebeat":{"events":{"active":0,"added":9,"done":9},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":1,"starts":1,"stops":0},"reloads":1,"scans":1},"output":{"events":{"acked":0,"active":0,"batches":1,"dropped":0,"duplicates":9,"failed":0,"toomany":0,"total":9},"read":{"bytes":11693,"errors":0},"type":"elasticsearch","write":{"bytes":28509,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":9,"retry":9,"total":9},"queue":{"acked":9,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":1},"load":{"1":0.1,"15":0.09,"5":0.12,"norm":{"1":0.1,"15":0.09,"5":0.12}}}}}}
2022-06-07T02:35:57.151Z        INFO    [monitoring]    log/log.go:193  Uptime: 26.952639902s
2022-06-07T02:35:57.151Z        INFO    [monitoring]    log/log.go:160  Stopping metrics logging.
2022-06-07T02:35:57.151Z        INFO    instance/beat.go:497    filebeat stopped.
[root@elastic filebeat]# nano /etc/filebeat/modules.d/threatintel.yml
[root@elastic filebeat]# filebeat -e
2022-06-07T02:36:28.065Z        INFO    instance/beat.go:685    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-06-07T02:36:28.065Z        INFO    instance/beat.go:693    Beat ID: 304c5e9c-7bda-4308-aaf2-bf8e67b93eaf
2022-06-07T02:36:28.067Z        WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: network is unreachable. No token in the metadata request will be used.
2022-06-07T02:36:28.067Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2022-06-07T02:36:28.067Z        INFO    [beat]  instance/beat.go:1039   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "304c5e9c-7bda-4308-aaf2-bf8e67b93eaf"}}}
2022-06-07T02:36:28.067Z        INFO    [beat]  instance/beat.go:1048   Build info      {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2022-06-07T02:36:28.067Z        INFO    [beat]  instance/beat.go:1051   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.17.9"}}}
2022-06-07T02:36:28.068Z        INFO    [beat]  instance/beat.go:1055   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-06-06T19:40:40Z","containerized":false,"name":"elastic.local","ip":["127.0.0.1/8","::1/128","10.0.0.101/24","fe80::a00:27ff:fe77:d8db/64","10.0.3.15/24","fe80::a00:27ff:fe77:a598/64"],"kernel_version":"4.18.0-383.el8.x86_64","mac":["08:00:27:77:d8:db","08:00:27:77:a5:98"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Stream","version":"8","major":8,"minor":0,"patch":0},"timezone":"UTC","timezone_offset_sec":0,"id":"a80ccfe94c474a9eb0c565871ef0ca0c"}}}
2022-06-07T02:36:28.068Z        INFO    [beat]  instance/beat.go:1084   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 4250, "ppid": 1682, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-06-07T02:36:27.890Z"}}}
2022-06-07T02:36:28.068Z        INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.4
2022-06-07T02:36:28.068Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.4' as ILM is enabled.
2022-06-07T02:36:28.068Z        INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: http://localhost:9200
2022-06-07T02:36:28.069Z        INFO    [publisher]     pipeline/module.go:113  Beat name: elastic.local
2022-06-07T02:36:28.070Z        INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2022-06-07T02:36:28.070Z        INFO    instance/beat.go:492    filebeat start running.
2022-06-07T02:36:28.070Z        INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2022-06-07T02:36:28.070Z        INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=11
2022-06-07T02:36:28.071Z        INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2022-06-07T02:36:28.071Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 1
2022-06-07T02:36:28.071Z        INFO    [crawler]       beater/crawler.go:117   starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]
2022-06-07T02:36:28.071Z        INFO    [crawler]       beater/crawler.go:121   input disabled, skipping it
2022-06-07T02:36:28.073Z        WARN    [input] v2/loader.go:104        BETA: The http_endpoint input is beta   {"input": "http_endpoint", "stability": "Beta", "deprecated": false}
2022-06-07T02:36:28.075Z        INFO    [crawler]       beater/crawler.go:106   Loading and starting Inputs completed. Enabled inputs: 0
2022-06-07T02:36:28.075Z        INFO    cfgfile/reload.go:164   Config reloader started
2022-06-07T02:36:28.080Z        WARN    [input] v2/loader.go:104        BETA: The http_endpoint input is beta   {"input": "http_endpoint", "stability": "Beta", "deprecated": false}
2022-06-07T02:36:28.081Z        INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: http://localhost:9200
2022-06-07T02:36:28.081Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2022-06-07T02:36:28.083Z        INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2022-06-07T02:36:28.089Z        INFO    cfgfile/reload.go:224   Loading of config files completed.
2022-06-07T02:36:28.089Z        INFO    [input.httpjson-stateless]      compat/compat.go:111    Input httpjson-stateless starting       {"id": "86F57B6043A0C821"}
2022-06-07T02:36:28.089Z        INFO    [input.httpjson-stateless]      v2/input.go:112 Process another repeated request.       {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2022-06-07T02:36:28.089Z        INFO    [input.httpjson-cursor] compat/compat.go:111    Input httpjson-cursor starting  {"id": "5B4CB7C8F10729D1"}
2022-06-07T02:36:28.089Z        INFO    [input.httpjson-cursor] compat/compat.go:111    Input httpjson-cursor starting  {"id": "99629170FE3D3B02"}
2022-06-07T02:36:28.089Z        INFO    [input.http_endpoint]   compat/compat.go:111    Input http_endpoint starting    {"id": "885D95D6F64FD52C"}
2022-06-07T02:36:28.089Z        INFO    [input.http_endpoint]   http_endpoint/input.go:121      Starting HTTP server on localhost:8080  {"id": "885D95D6F64FD52C", "address": "localhost:8080"}
2022-06-07T02:36:28.090Z        INFO    [input.httpjson-cursor] v2/input.go:112 Process another repeated request.       {"id": "5B4CB7C8F10729D1", "input_source": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc", "input_url": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc"}
2022-06-07T02:36:28.091Z        INFO    [input.httpjson-cursor] v2/input.go:112 Process another repeated request.       {"id": "99629170FE3D3B02", "input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects"}
2022-06-07T02:36:28.310Z        ERROR   [input.httpjson-cursor] v2/input.go:115 Error while processing http request: failed to execute http client.Do: server responded with status code 401: {"error":{"status":401}}                                                                                          {"id": "5B4CB7C8F10729D1", "input_source": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc", "input_url": "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false&orderby=lastseen&direction=asc"}
2022-06-07T02:36:28.741Z        INFO    [input.httpjson-stateless]      v2/request.go:204       request finished: 9 events published    {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2022-06-07T02:36:29.267Z        INFO    [input.httpjson-cursor] v2/request.go:204       request finished: 0 events published    {"id": "99629170FE3D3B02", "input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects"}
2022-06-07T02:36:29.741Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(http://localhost:9200))
2022-06-07T02:36:29.742Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-06-07T02:36:29.742Z        INFO    [publisher]     pipeline/retry.go:223     done
2022-06-07T02:36:29.744Z        INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2022-06-07T02:36:29.762Z        INFO    [esclientleg]   eslegclient/connection.go:285   Attempting to connect to Elasticsearch version 7.17.4
2022-06-07T02:36:29.784Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2022-06-07T02:36:29.798Z        INFO    [index-management.ilm]  ilm/std.go:170  ILM policy filebeat exists already.
2022-06-07T02:36:29.798Z        INFO    [index-management]      idxmgmt/std.go:397      Set setup.template.name to '{filebeat-7.17.4 {now/d}-000001}' as ILM is enabled.
2022-06-07T02:36:29.798Z        INFO    [index-management]      idxmgmt/std.go:402      Set setup.template.pattern to 'filebeat-7.17.4-*' as ILM is enabled.
2022-06-07T02:36:29.798Z        INFO    [index-management]      idxmgmt/std.go:436      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.4 {now/d}-000001} as ILM is enabled.
2022-06-07T02:36:29.798Z        INFO    [index-management]      idxmgmt/std.go:440      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-06-07T02:36:29.809Z        INFO    template/load.go:110    Template "filebeat-7.17.4" already exists and will not be overwritten.
2022-06-07T02:36:29.809Z        INFO    [index-management]      idxmgmt/std.go:297      Loaded index template.
2022-06-07T02:36:29.811Z        INFO    [index-management.ilm]  ilm/std.go:126  Index Alias filebeat-7.17.4 exists already.
2022-06-07T02:36:29.811Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(http://localhost:9200)) established

Charles_Allen · June 7, 2022, 2:40am

The following lines seem relevant.

2022-06-07T02:36:28.741Z        INFO    [input.httpjson-stateless]      v2/request.go:204       request finished: 9 events published    {"id": "86F57B6043A0C821", "input_url": "https://mb-api.abuse.ch/api/v1/"}
2022-06-07T02:36:29.267Z        INFO    [input.httpjson-cursor] v2/request.go:204       request finished: 0 events published    {"id": "99629170FE3D3B02", "input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects"}

How do I make it publish events?

system · July 5, 2022, 2:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.