AWS through Proxy

[spike83]

Hi,

How can I add a proxy setting the the AWS module to get to the outside world? I appear to run into issues using the system variable https_proxy when outputting to elastic internally, I can either get to AWS and not Elastic, or to Elastic and not AWS

Regards

[jsoriano (Jaime Soriano)]

Hi @spike83,

Where https_proxy works, you should be also able of using the no_proxy environment variable. You can set this variable to a comma separated list of hosts that shouldn't use proxy.

See the golang docs about this: https://golang.org/pkg/net/http/#ProxyFromEnvironment

[jsoriano (Jaime Soriano)]

More info here about the values accepted by no_proxy https://godoc.org/golang.org/x/net/http/httpproxy#Config

[spike83]

Thank you.

I have sorted my proxy issue out now, but.... I can run the modules test command and connect to my AWS account, but Metricbeat running as a service times out connecting.

[aws.ebs]#011aws/aws.go:93#011failed to list account aliases, please check permission setting: RequestError: send request failed
metricbeat: caused by: Post https://iam.amazonaws.com/: dial tcp 54.239.22.207:443: connect: connection timed out

[jsoriano (Jaime Soriano)]

Umm, this is weird, if metricbeat test can connect, then metricbeat should be also able to connect. Is it possible that they are being executed with different environment? Do you use some service manager like systemd to run metricbeat?

[spike83]

Further testing has shown that if I run metricbeat manually with the following parameters

metricbeat -e -d "*" >> log_file.txt 2>&1

I can see data being collected and it displays in my dashboard

If I start metricbeat using systemctl it fails to collect the data from aws

[jsoriano (Jaime Soriano)]

Yeah, this is expected. When you run a service in systemd it starts with a clear environment. If you want to add environment variables you have to add them to the systemd configuration. You can do it for this case by adding a drop-in file, for example in /etc/systemd/system/metricbeat.service.d/proxy.conf, with a content like the following one:

[Service]
Environment="https_proxy=..."
Environment="no_proxy=..."

Then systemctl daemon-reload and systemctl restart metricbeat.

You can read more about customizing Systemd configuration for metricbeat here: https://www.elastic.co/guide/en/beats/metricbeat/7.6/running-with-systemd.html#_customize_systemd_unit_for_metricbeat

[spike83]

Thank you Jaime, appreciate the help