Question regarding specifying "localhost" and deploying to Lambda

Elastic Stack Beats
1

Running on my localhost (i.e. my laptop):
ES 7.3 w. Basic-license (port 9200 is verified and returns info)
Kibana 7.3 w. Basic-license (port 5601 is verified and returns the Kibana interface)
Functionbeat 7.3 from the link provided when starting Kibana;

^ The hash when downloading either through selecting "OSS" or "Basic" is the same

My "functionbeat.yml";

functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
functionbeat.provider.aws.deploy_bucket: "mybucket"
functionbeat.provider.aws.functions:
  - name: cloudwatch-logs
    enabled: true
    type: cloudwatch_logs
    triggers:
      - log_group_name: myloggroup

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "changeme"
setup.kibana:
  host: "localhost:5601"

Deploying to Lambda goes fine, I change the timeout of the Lambda which defaults to 3sec to 1min30sec and up the RAM to 512 but then when I try and "Check data" in Kibana it says "No data has been received from Functionbeat yet".

When I check the CloudWatch-logs for the Lambda I see this;
INFO [license-manager] licenser/manager.go:265 Cannot retrieve license, retrying later, error: Get http://localhost:9200: dial tcp [::1]:9200: socket: address family not supported by protocol

Since I specified "localhost" which makes sense on my local computer I guess I can't specify it as "localhost" when I upload it as a Lambda?
How does Lambda know what my "localhost" was, i.e. "localhost" is always "localhost" and dependent on the host you run it on and the actual IP varies depending on host, or am I missing something?

I haven't started Functionbeat locally except for running;
.\functionbeat.exe setup
.\functionbeat.exe deploy cloudwatch-logs

AFAIK it creates it as a Lambda and Functionbeat does not need to be started on your "localhost" (i.e. my laptop)?

Sorry if I've missed something, great tutorial when starting Kibana and going to "Add log data" > "CloudWatch Logs" just think I misunderstand something.

Best Regards & Thanks in advance - TheSwede86

2

The lambda function gets deployed on AWS, it is not running locally on your machine, and once depoyed it doesn't know about your laptop. You need an Elasticsearch running in the Cloud, like Elastic Cloud.

3

Thank you for your reply.

What I did to solve it to have the following in my Functionbeat.yml:

The " security_group_ids" is the SecurityGroupID of the SG I created in advance that the Lambda will use and "role" is the IAM-role we use for our Lambdas which needed some added permissions since we normally don't deploy Lambdas in a VPC. The IP-specified is the internal IP of the host running ES and Kibana.

The ES and Kibana host are the internal IP of the server hosting those applications and on that server I needed to edit the following;

...\elasticsearch\config\elasticsearch.yml;

cluster.name: myEScluster
node.name: myHostname
network.host: internal-ipOfhost
cluster.initial_master_nodes: ["myHostname"]

---\kibana\config\kibana.yml

server.host: "internal-ipOfhost"
elasticsearch.hosts: ["http://internal-ipOfhost:9200]

That should be it, hopefully it helps someone :slight_smile:

2 Likes
4

Thank you for sharing the details of your solution!

5

This is strange and contrary, to my post here

My ES is not exposed to internet but when I run ./functionbeat im getting messages on my cli?

6

So in that case if ES is not routable, when you running ./functionbeat there should be no msgs on CLI?

pls see this

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.