Cannot retrieve license. Connection refused

jbuenostein89 · August 30, 2019, 5:31pm

Getting an error while trying to retrieve the license. I confirmed in Kibana that we have an active basic license. We are running everything locally.

2019-08-30T17:28:10.007Z        DEBUG   [license-manager]       licenser/manager.go:274 License is too old, grace time remaining: 44m51.049174851s
2019-08-30T17:28:22.214Z        DEBUG   [elasticsearch] elasticsearch/client.go:719     ES Ping(url=http://172.18.2.76:9200)
2019-08-30T17:28:22.214Z        DEBUG   [elasticsearch] elasticsearch/client.go:723     Ping request failed with: Get http://172.18.2.76:9200: dial tcp 172.18.2.76:9200: connect: connection refused
2019-08-30T17:28:22.214Z        INFO    [license-manager]       licenser/manager.go:265 Cannot retrieve license, retrying later, error: Get http://172.18.2.76:9200: dial tcp 172.18.2.76:9200: connect: connection refused
could not retrieve the license information from the cluster
github.com/elastic/beats/x-pack/libbeat/licenser.(*ElasticFetcher).Fetch
        /go/src/github.com/elastic/beats/x-pack/libbeat/licenser/elastic_fetcher.go:128
github.com/elastic/beats/x-pack/libbeat/licenser.(*Manager).update
        /go/src/github.com/elastic/beats/x-pack/libbeat/licenser/manager.go:263
github.com/elastic/beats/x-pack/libbeat/licenser.(*Manager).worker
        /go/src/github.com/elastic/beats/x-pack/libbeat/licenser/manager.go:241
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337

Christian_Dahlqvist · September 1, 2019, 6:32pm

Can you check there is no firewall blocking access?

jbuenostein89 · September 1, 2019, 11:38pm

No firewall is blocking communication. Everything is running locally on the same server and no firewall has been configured.

stephenb · September 2, 2019, 12:20am

What is the result if you run the get license API from Dev tools

GET /_license

jbuenostein89 · September 5, 2019, 2:42pm

Result of GET /_license:

{
  "license" : {
    "status" : "active",
    "uid" : "713dd89c-e857-4a0a-a48d-feeeec673c35",
    "type" : "basic",
    "issue_date" : "2019-08-15T18:37:37.311Z",
    "issue_date_in_millis" : 1565894257311,
    "max_nodes" : 1000,
    "issued_to" : "elasticsearch",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

stephenb · September 5, 2019, 3:12pm

I think there is still a connectivity / config issue.

Is that log from kibana log? you indicate you're running everything locally. When you look at the elasticsearch logs does everything look correct ... No errors?

if everything is running locally you should not have had to put in any IP addresses in the config files they all would have defaulted to localhost
And elasticsearch and kibana would all run on localhost and find each other automatically

So from the cmd line run

curl http://172.18.2.76:9200

jbuenostein89 · September 5, 2019, 3:42pm

So everything is running on the same system, but in AWS. It is deployed in a private subnet, so only has a private IP. When I originally had localhost in the config files, I was not able to access Kibana. The log in my original post is from running functionbeat -e -d "*".

kibana.yml has the following set:
server.host: "172.18.2.76" << when I set this to localhost, I cannot access kibana
elasticsearch.hosts: ["http://localhost:9200"]

Everything in elasticsearch.yml is the default.

curl http://172.18.2.76:9200 returns:
curl: (7) Failed to connect to 172.18.2.76 port 9200: Connection refused

curl http://localhost:9200 returns:

    {
      "name" : "ip-172-18-2-76",
      "cluster_name" : "elasticsearch",
      "cluster_uuid" : "-aTkeTgzT-66zr-p3o6Dsw",
      "version" : {
        "number" : "7.3.0",
        "build_flavor" : "default",
        "build_type" : "deb",
        "build_hash" : "de777fa",
        "build_date" : "2019-07-24T18:30:11.767338Z",
        "build_snapshot" : false,
        "lucene_version" : "8.1.0",
        "minimum_wire_compatibility_version" : "6.8.0",
        "minimum_index_compatibility_version" : "6.0.0-beta1"
      },
      "tagline" : "You Know, for Search"
    }

stephenb · September 5, 2019, 4:24pm

Assuming this is a single node?....

In elasticsearch.yml

network.host: 172.18.2.76

and

discovery.type: single-node

Please read about this in the following pages as it is important to understand the what and why, In short you are now making Elasticsearch network addressable... and the implications that come with that

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/network.host.html

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/system-config.html#dev-vs-prod

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/bootstrap-checks.html#single-node-discovery

I answered a little more about this here

jbuenostein89 · September 5, 2019, 5:49pm

Thanks for those links. I wasn't aware of the discovery. type bit.

I made those changes, saved, restarted the elasticsearch service and it looks like we're making some progress:

root@ip-172-18-2-76:/home/ubuntu/functionbeat-7.3.0-linux-x86_64# curl http://172.18.2.76:9200
{
  "name" : "ip-172-18-2-76",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "-aTkeTgzT-66zr-p3o6Dsw",
  "version" : {
    "number" : "7.3.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "de777fa",
    "build_date" : "2019-07-24T18:30:11.767338Z",
    "build_snapshot" : false,
    "lucene_version" : "8.1.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

When I run functionbeat -e -d "*" I now get the following (see last line for error):

2019-09-05T17:37:47.429Z        INFO    instance/beat.go:292    Setup Beat: functionbeat; Version: 7.3.0
2019-09-05T17:37:47.429Z        DEBUG   [beat]  instance/beat.go:318    Initializing output plugins
2019-09-05T17:37:47.429Z        INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'functionbeat-7.3.0' as ILM is enabled.
2019-09-05T17:37:47.429Z        INFO    elasticsearch/client.go:170     Elasticsearch url: http://172.18.2.76:9200
2019-09-05T17:37:47.429Z        DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2019-09-05T17:37:47.429Z        INFO    [publisher]     pipeline/module.go:97   Beat name: ip-172-18-2-76
2019-09-05T17:37:47.429Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-09-05T17:37:47.429Z        INFO    instance/beat.go:421    functionbeat start running.
2019-09-05T17:37:47.429Z        INFO    [functionbeat]  beater/functionbeat.go:74       Functionbeat is running
2019-09-05T17:37:47.429Z        INFO    elasticsearch/client.go:170     Elasticsearch url: http://172.18.2.76:9200
2019-09-05T17:37:47.430Z        INFO    [license-manager]       licenser/manager.go:184 License manager started, retrieving initial license
2019-09-05T17:37:47.430Z        INFO    [functionbeat]  licenser/manager.go:331 Waiting on synchronous license check
2019-09-05T17:37:47.430Z        DEBUG   [license-manager]       licenser/manager.go:230 Starting periodic license check, refresh: 15m0s grace: 45m0s
2019-09-05T17:37:48.085Z        DEBUG   [elasticsearch] elasticsearch/client.go:719     ES Ping(url=http://172.18.2.76:9200)
2019-09-05T17:37:48.089Z        DEBUG   [elasticsearch] elasticsearch/client.go:742     Ping status code: 200
2019-09-05T17:37:48.089Z        INFO    elasticsearch/client.go:743     Attempting to connect to Elasticsearch version 7.3.0
2019-09-05T17:37:48.089Z        DEBUG   [elasticsearch] elasticsearch/client.go:761     GET http://172.18.2.76:9200/_xpack?human=false  <nil>
2019-09-05T17:37:48.108Z        INFO    [license-manager]       licenser/manager.go:282 Valid license retrieved                                                                                                {"license mode": "Basic", "type": "Basic", "status": "Active"}
2019-09-05T17:37:48.108Z        DEBUG   [license-manager]       licenser/manager.go:319 License information updated
2019-09-05T17:37:48.108Z        DEBUG   [license-manager]       licenser/manager.go:222 Notifying 1 watchers
2019-09-05T17:37:48.108Z        DEBUG   [functionbeat]  licenser/manager.go:334 Validating license
2019-09-05T17:37:48.108Z        DEBUG   [functionbeat]  licenser/check.go:31    Checking that license covers %sBasic
2019-09-05T17:37:48.108Z        INFO    [functionbeat]  licenser/manager.go:339 License is valid, mode: Basic
2019-09-05T17:37:48.108Z        INFO    [functionbeat]  beater/functionbeat.go:92       Functionbeat is configuring enabled functions: 
2019-09-05T17:37:48.108Z        DEBUG   [license-manager]       licenser/manager.go:247 Periodic license check is stopped
2019-09-05T17:37:48.108Z        DEBUG   [license-manager]       licenser/manager.go:218 No watchers configured
2019-09-05T17:37:48.108Z        INFO    [license-manager]       licenser/manager.go:211 License manager stopped
2019-09-05T17:37:48.108Z        INFO    [functionbeat]  beater/functionbeat.go:96       Functionbeat stopped running
2019-09-05T17:37:48.112Z        INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":16}},"total":{"ticks":30,"time":{"ms":37},"value":30},"user":{"ticks":20,"time":{"ms":21}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":6},"info":{"ephemeral_id":"d8533474-4226-478b-8413-2165f145edca","uptime":{"ms":696}},"memstats":{"gc_next":4194304,"memory_alloc":2564144,"memory_total":4689832,"rss":29523968},"runtime":{"goroutines":11}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":0.04,"15":0.04,"5":0.08,"norm":{"1":0.01,"15":0.01,"5":0.02}}}}}}
2019-09-05T17:37:48.112Z        INFO    [monitoring]    log/log.go:154  Uptime: 697.524297ms
2019-09-05T17:37:48.112Z        INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2019-09-05T17:37:48.112Z        INFO    instance/beat.go:431    functionbeat stopped.
2019-09-05T17:37:48.112Z        ERROR   instance/beat.go:877    Exiting: error when creating the functions, error: no function are enabled for selected provider: 'aws'
Exiting: error when creating the functions, error: no function are enabled for selected provider: 'aws'

Not sure if that error would warrant another thread, but I believe everything was deployed successfully in AWS. The deployment logs in CloudFormation had no errors and the Lambda seems to be working as well.

I have not seen this error until now.

Edit:
This is my functionbeat.yml:

functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
functionbeat.provider.aws.deploy_bucket: "wazuh-functionbeat-poc-00000151"
functionbeat.provider.aws.functions:  
- name: cloudwatch    
  enabled: true
   type: cloudwatch_logs
   triggers:
        - log_group_name: /aws/events/ec2-state-change
    description: "lambda function for cloudwatch logs"

stephenb · September 5, 2019, 6:02pm

Personally I would open a separate thread with with the proper title etc. You will get a better chance of getting your questions answered. I am not a functionbeat wiz...

jbuenostein89 · September 5, 2019, 6:20pm

Great, well thanks for your help!!

system · September 26, 2019, 8:20am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.