Elastic Cloud Output not working Tshoot

Leon21 · April 17, 2021, 8:36pm

Hi
I'm trying to push my events from a filebeat module into elastic cloud but could not find them into ES in my filebeat* indexes.

filebeat module gets events from a tcp socket (filebeat listen to it successfully) and has a pipeline and the elastic cloud output.

here is the result when I send an event to the tcp socket that the filebeat is listening:

2021-04-17T20:11:31.222Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://125f4af39bc74494afexxxxxxxxxxxxxxx.us-west1.gcp.cloud.es.io:443))
2021-04-17T20:11:31.223Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2021-04-17T20:11:31.223Z        INFO    [publisher]     pipeline/retry.go:223     done
2021-04-17T20:11:31.915Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.12.0
2021-04-17T20:11:32.061Z        INFO    [license]       licenser/es_callback.go:51      Elasticsearch license: Platinum
2021-04-17T20:11:32.478Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.12.0
2021-04-17T20:11:32.760Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2021-04-17T20:11:32.899Z        INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2021-04-17T20:11:32.899Z        INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2021-04-17T20:11:32.899Z        INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{filebeat-7.12.0 {now/d}-000001}' as ILM is enabled.
2021-04-17T20:11:32.899Z        INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'filebeat-7.12.0-*' as ILM is enabled.
2021-04-17T20:11:32.899Z        INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.12.0 {now/d}-000001} as ILM is enabled.
2021-04-17T20:11:32.899Z        INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-17T20:11:33.038Z        INFO    template/load.go:97     Template filebeat-7.12.0 already exists and will not be overwritten.
2021-04-17T20:11:33.038Z        INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2021-04-17T20:11:33.177Z        INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
2021-04-17T20:11:33.177Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://125f4af39bc74494afexxxxxxxxxxxxxxxxx.us-west1.gcp.cloud.es.io:443)) established

I get this only once for the first event only but no error or nothing.

at Kibana I check the events under Discover but it seems nothing is coming from filebeat.

Module Input:

type: tcp
host: "localhost:10000"
max_message_size: 10MiB
#framing: rfc6587

tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
  - decode_json_fields:
      fields:
        - message
      target: json
{{ if eq .keep_original_message true }}
  - rename:
      fields:
        - from: message
          to: event.original
{{ end }}
  - add_fields:
      target: ''
      fields:
        ecs.version: 1.8.0

The point is when I change the output from elastic cloud to console I could see the events but when I switch it back to elastic cloud nothing is working.

I want to troubleshoot but I don't know what should I check.
appreciate if you could help
Thanks

warkolm · April 20, 2021, 2:22am

Can you try running Filebeat with debug enabled so we can see a bit more info?

Leon21 · April 25, 2021, 10:28pm

Hey Mark @warkolm
Sorry for delay
here is the debug enabled output
it seems the event is successfully published to ES. but I could not view it in either in dashboards or via error.message in Discover panel (for ingest parsing errors).

Is there any place to look for it?
thanks

2021-04-25T22:01:36.725Z        DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:128     add_cloud_metadata: starting to fetch metadata, timeout=3s
2021-04-25T22:01:36.888Z        DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:165     add_cloud_metadata: received disposition for digitalocean after 162.679472ms. result=[provider:digitalocean, error=failed to unmarshal digitalocean JSON of '': EOF, metadata={}]
2021-04-25T22:01:36.888Z        DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:165     add_cloud_metadata: received disposition for aws after 162.822299ms. result=[provider:aws, error=failed to unmarshal aws JSON of '': EOF, metadata={}]
2021-04-25T22:01:36.895Z        DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:165     add_cloud_metadata: received disposition for azure after 169.973871ms. result=[provider:azure, error=failed to unmarshal azure JSON of '': EOF, metadata={}]
2021-04-25T22:01:36.906Z        DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:165     add_cloud_metadata: received disposition for gcp after 181.216557ms. result=[provider:gcp, error=failed to unmarshal gcp JSON of '': EOF, metadata={}]
2021-04-25T22:01:37.342Z        INFO    instance/beat.go:304    Setup Beat: filebeat; Version: 7.12.0
2021-04-25T22:01:37.342Z        DEBUG   [beat]  instance/beat.go:330    Initializing output plugins
2021-04-25T22:01:37.342Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.12.0' as ILM is enabled.
2021-04-25T22:01:37.343Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443
2021-04-25T22:01:37.343Z        DEBUG   [publisher]     pipeline/consumer.go:148        start pipeline event consumer
2021-04-25T22:01:37.343Z        INFO    [publisher]     pipeline/module.go:113  Beat name: servername
2021-04-25T22:01:37.363Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-04-25T22:01:37.363Z        INFO    instance/beat.go:468    filebeat start running.
2021-04-25T22:01:37.364Z        INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2021-04-25T22:01:37.364Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 2
2021-04-25T22:01:37.364Z        DEBUG   [cfgfile]       cfgfile/reload.go:132   Checking module configs from: /root/temp-fb/filebeat/modules.d/*.yml
2021-04-25T22:01:37.365Z        DEBUG   [registrar]     registrar/registrar.go:140      Starting Registrar
2021-04-25T22:01:37.365Z        DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /root/temp-fb/filebeat/modules.d/okta.yml
2021-04-25T22:01:37.365Z        DEBUG   [cfgfile]       cfgfile/reload.go:146   Number of module configs found: 1
2021-04-25T22:01:37.368Z        INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 0
2021-04-25T22:01:37.368Z        INFO    cfgfile/reload.go:164   Config reloader started
2021-04-25T22:01:37.368Z        DEBUG   [cfgfile]       cfgfile/reload.go:194   Scan for new config files
2021-04-25T22:01:37.368Z        DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /root/temp-fb/filebeat/modules.d/okta.yml
2021-04-25T22:01:37.368Z        DEBUG   [cfgfile]       cfgfile/reload.go:213   Number of module configs found: 1
2021-04-25T22:01:37.368Z        DEBUG   [reload]        cfgfile/list.go:63      Starting reload procedure, current runners: 0
2021-04-25T22:01:37.368Z        DEBUG   [reload]        cfgfile/list.go:81      Start list: 1, Stop list: 0
2021-04-25T22:01:37.370Z        DEBUG   [processors]    processors/processor.go:120     Generated new processors: decode_json_fields=message, rename=[{From:message To:event.original}], add_fields={"ecs":{"version":"1.8.0"}}
2021-04-25T22:01:37.370Z        DEBUG   [reload]        cfgfile/list.go:105     Starting runner: okta (system)
2021-04-25T22:01:37.370Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443
2021-04-25T22:01:37.370Z        DEBUG   [esclientleg]   eslegclient/connection.go:290   ES Ping(url=https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443)
2021-04-25T22:01:37.933Z        DEBUG   [esclientleg]   eslegclient/connection.go:313   Ping status code: 200
2021-04-25T22:01:37.933Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.12.0
2021-04-25T22:01:37.933Z        DEBUG   [modules]       fileset/pipelines.go:67 Required processors: [{geoip ingest-geoip}]
2021-04-25T22:01:37.933Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_nodes/ingest  <nil>
2021-04-25T22:01:38.033Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_ingest/pipeline/filebeat-7.12.0-okta-system-pipeline  <nil>
2021-04-25T22:01:38.137Z        DEBUG   [modules]       fileset/pipelines.go:120        Pipeline already exists in Elasticsearch.       {"pipeline": "filebeat-7.12.0-okta-system-pipeline"}
2021-04-25T22:01:38.137Z        INFO    cfgfile/reload.go:224   Loading of config files completed.
2021-04-25T22:01:38.137Z        INFO    [input.tcp]     tcp/input.go:105        Starting TCP input      {"address": "localhost:9100"}
2021-04-25T22:01:38.138Z        INFO    [tcp]   streaming/listener.go:120       Started listening for TCP connection    {"address": "localhost:9100"}
2021-04-25T22:01:48.138Z        DEBUG   [input] input/input.go:139      Run input
2021-04-25T22:01:58.139Z        DEBUG   [input] input/input.go:139      Run input
2021-04-25T22:01:58.454Z        DEBUG   [tcp]   streaming/listener.go:149       New client      {"address": "localhost:9100", "remote_address": "127.0.0.1:56894", "total": 1}
2021-04-25T22:01:58.466Z        DEBUG   [processors]    processing/processors.go:203    Publish event: {
  "@timestamp": "2021-04-25T22:01:58.465Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.0",
    "pipeline": "filebeat-7.12.0-okta-system-pipeline"
  },
  "tags": [
    "forwarded"
  ],
  "fileset": {
    "name": "system"
  },
  "json": {
    // my event is here
    },
  "event": {
    "module": "okta",
    "dataset": "okta.system",
    "original": "{

    //my original event is here
    }"
  },
  "service": {
    "type": "okta"
  },
  "cloud": {
    "availability_zone": "",
    "provider": "openstack",
    "machine": {
      "type": ""
    },
    "instance": {
      "name": "servername",
      "id": "c00dfcfc6faa4b2aaeba802xxxxxxxxx"
    }
  }
}
2021-04-25T22:01:58.466Z        DEBUG   [tcp]   streaming/listener.go:163       client disconnected     {"address": "localhost:9100", "remote_address": "127.0.0.1:56894", "total": 1}
2021-04-25T22:01:59.467Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443))
2021-04-25T22:01:59.467Z        DEBUG   [esclientleg]   eslegclient/connection.go:290   ES Ping(url=https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443)
2021-04-25T22:01:59.468Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2021-04-25T22:01:59.468Z        INFO    [publisher]     pipeline/retry.go:223     done
2021-04-25T22:01:59.849Z        DEBUG   [esclientleg]   eslegclient/connection.go:313   Ping status code: 200
2021-04-25T22:01:59.849Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.12.0
2021-04-25T22:01:59.849Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_license?human=false  <nil>
2021-04-25T22:01:59.939Z        DEBUG   [license]       licenser/check.go:31    Checking that license covers %sBasic
2021-04-25T22:01:59.939Z        INFO    [license]       licenser/es_callback.go:51      Elasticsearch license: Platinum
2021-04-25T22:01:59.939Z        DEBUG   [modules]       fileset/pipelines.go:67 Required processors: [{geoip ingest-geoip}]
2021-04-25T22:01:59.939Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_nodes/ingest  <nil>
2021-04-25T22:02:00.028Z        DEBUG   [esclientleg]   eslegclient/connection.go:290   ES Ping(url=https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443)
2021-04-25T22:02:00.116Z        DEBUG   [esclientleg]   eslegclient/connection.go:313   Ping status code: 200
2021-04-25T22:02:00.116Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.12.0
2021-04-25T22:02:00.119Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_ingest/pipeline/filebeat-7.12.0-okta-system-pipeline  <nil>
2021-04-25T22:02:00.209Z        DEBUG   [modules]       fileset/pipelines.go:120        Pipeline already exists in Elasticsearch.       {"pipeline": "filebeat-7.12.0-okta-system-pipeline"}
2021-04-25T22:02:00.209Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_xpack  <nil>
2021-04-25T22:02:00.297Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2021-04-25T22:02:00.297Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_ilm/policy/filebeat  <nil>
2021-04-25T22:02:00.386Z        INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2021-04-25T22:02:00.386Z        INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2021-04-25T22:02:00.386Z        INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{filebeat-7.12.0 {now/d}-000001}' as ILM is enabled.
2021-04-25T22:02:00.387Z        INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'filebeat-7.12.0-*' as ILM is enabled.
2021-04-25T22:02:00.387Z        INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.12.0 {now/d}-000001} as ILM is enabled.
2021-04-25T22:02:00.387Z        INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-25T22:02:00.387Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_cat/templates/filebeat-7.12.0  <nil>
2021-04-25T22:02:00.494Z        INFO    template/load.go:97     Template filebeat-7.12.0 already exists and will not be overwritten.
2021-04-25T22:02:00.494Z        INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2021-04-25T22:02:00.494Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/_alias/filebeat-7.12.0  <nil>
2021-04-25T22:02:00.585Z        INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
2021-04-25T22:02:00.585Z        DEBUG   [esclientleg]   eslegclient/connection.go:364   GET https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443/  <nil>
2021-04-25T22:02:00.672Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://60caa01658d945f7863xxxxxxxxxxxxx.us-east-1.aws.found.io:443)) established
2021-04-25T22:02:00.772Z        DEBUG   [elasticsearch] elasticsearch/client.go:230     PublishEvents: 1 events have been published to elasticsearch in 99.938957ms.
2021-04-25T22:02:00.772Z        DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 1]
2021-04-25T22:02:00.772Z        DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=1, start-seq=1, end-seq=1
2021-04-25T22:02:00.772Z        DEBUG   [acker] beater/acker.go:64      stateless ack   {"count": 1}
2021-04-25T22:02:00.773Z        DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2021-04-25T22:02:00.773Z        DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack

warkolm · April 26, 2021, 5:12am

That suggests something has been indexed?

Leon21 · April 26, 2021, 4:11pm

Yes, it should be indexed but when I search filebeat-* for any events (this is clean ES Cloud Install) I saw nothing. that's weird!

Thanks again for your rapid response

Leon21 · April 26, 2021, 4:29pm

Just an Update.
I could see one document is updated into the index but still I could not search it (Both via Discover or available dashboards)

system · May 24, 2021, 6:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.