# Collect logs from external deployments exposing a domain **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [July 15, 2022, 2:15pm UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746 "2022-07-15T14:15:37Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![thahgr](https://avatars.discourse-cdn.com/v4/letter/t/e19b73/32.png) [@thahgr](https://discuss.elastic.co/u/thahgr)\ **Post date:** [July 15, 2022, 2:15pm UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746/1 "2022-07-15T14:15:37Z")
TL;DR I want to ship logs from remote deployments via filebeat or logstash to a central Elastic/Elastic stack via HTTP exposing just domains I have considered two solutions, cant get any to work The setup is as follows: - there is a central big framework on the company's server A, - and multiple clients services each one in a different server B,C,D etc. **NONE** is in the same network. All services are on docker containers with compose. We set up Elastic Stack on A. The initial plan was to add filebeat on each server B C D and ship to logstash-elastic on A However A sits behind a company firewall and haproxy etc. They are able to open a domain and subdomains and map to a specific ports. For elastic and logstash [https://elastic.mycompany.com](https://elastic.mycompany.com) : 9200 [https://logstash.mycompany.com](https://logstash.mycompany.com) : 5044 1. The plan to ship logs with filebeat doesnt work, I guess because it can only connect via TCP and not HTTP The following two tries fail ```auto output.logstash: enabled: true hosts: ["https://logstash.mycompany.com"] ``` `Failed to connect to backoff(async(tcp://https://logstash.mycompany.com)): lookup https on 127.0.0.28:53: no such host` and setting just the domain it adds the **5044** ```auto output.logstash: enabled: true hosts: ["logstash.mycompany.com"] ``` `Failed to connect to backoff(async(tcp://logstash.mycompany.com):5044)): dial tcp [2606:asdfasdf::asdfasdf::]:5044: connect: network is unreachable` Going to plan 2 1. Deploy filebeat and logstash in each server B C D and ship to elastic on A The following setup doesnt work, the 9200 is added after the domain logstash.yml (on servers B C D) ```auto output { elasticsearch { hosts => ["https://elastic.mycompany.com"] index => "logs-%{+YYYY.MM.dd}" document_type => "nginx_logs" user => "elastic" password => "changeme" } stdout { codec => rubydebug } } ``` `][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elastic.mycompany.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elastic.mycompany.com:9200/][Manticore::ResolutionFailure] elastic.mycompany.com"}` **I am thinking another solution, maybe use the http plugin and send via REST requests ? I need some help with this configuration, what I have until now is the following (not working)** ```auto output { http { url => "https://elastic.mycompany.com" http_method => "post" content_type => "text/xml;charset=UTF-8" index => "logs-%{+YYYY.MM.dd}" headers => { "Authorization" => "Basic ZWxhc3RpYzpjaGFuZ2VtZQ==" } format => "message" } } ``` Thank you for your help! ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 15, 2022, 2:50pm UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746/2 "2022-07-15T14:50:59Z")
Is your `logstash.mycompany.com` listening on port `5044` on the internet or it is using another port, like `443` and redirecting it to `5044`? Filebeat does not use `http`, it uses a custom protocol over `tcp`, also your haproxy should be configured to use the `tcp` not `http`. Also, can you test if you can connect using telnet to both `logstash.mycompany.com` and `elastic.mycompany.com` from the external server? Your issues seems to be related to some network configuration, it won't work until you fix it, changing the output plugin will make no difference if it still cannot connect. ---
**Author:** ![thahgr](https://avatars.discourse-cdn.com/v4/letter/t/e19b73/32.png) [@thahgr](https://discuss.elastic.co/u/thahgr)\ **Post date:** [July 18, 2022, 6:58am UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746/3 "2022-07-18T06:58:02Z")
@leandrojmp Thank you for your reply, I dont about the first question, whether its 443 or 5044. The only clue I can give about this, is that if I start a hello world webservice on Server A listening to IP:port 192.168.x.x:5044 instead of logstash, I can access it outside the VPN network from [logstash.mycompany.com](http://logstash.mycompany.com) successfully. Same for 9200 and the elastic url This means the connection is active and correct. Not sure how its done internally though 🙂 So the question is how can I use the http plugin with this domain? when I enter it on the configuration I get results like `elastic.mycompany.com:9200/` where the 9200 is appended in the end. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 19, 2022, 4:26am UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746/4 "2022-07-19T04:26:25Z")
Never used the the `http` output to send data to elasticsearch, so I can not help with this, but it is not that simple as you would need to use the path for the index direct in the `url` option. Something like: `url => "https://elastic.mycompany.com/index-name/_doc` for example. But again, It still looks like you have a connectivity issue and needs to troubleshoot it before anything else. if the `elasticsearch` output cannot reach your server why would the `http` output reach it, since it is the same endpoint? Also, your elasticsearch error seems to indicate a DNS resolution failure. > [Manticore::ResolutionFailure] [elastic.mycompany.com](http://elastic.mycompany.com) Can you check what is really exposed in the internet? Is `logstash.mycompany.com` exposed in the public internet on port `5044` using TCP, not just HTTP/HTTPS, since it is behind a firewall? Can you from one of the remote machines connect to this domain in this port using telnet for example? Is `elasticsearch.mycompany.com` exposed in the public internet on port `9200` using `https`? Can you run a `curl https://elastic.mycompany.com/` from the Logstash server and get a response? ---
**Author:** ![thahgr](https://avatars.discourse-cdn.com/v4/letter/t/e19b73/32.png) [@thahgr](https://discuss.elastic.co/u/thahgr)\ **Post date:** [July 19, 2022, 7:33am UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746/5 "2022-07-19T07:33:48Z")
There is no issue with the domains, its just like any domain exposing an ip port. As I said the hello world tries work fine. I was finally able to find a solution. For anyone that may bump on this question in the future. Logstash to elastic and beats to logstash are done with tcp protocols thus in our setup it cannot be done. The solution is to use logstash to logstash, http output to http input plugin as explained in the official documentation [Logstash-to-Logstash communication | Logstash Reference [8.3] | Elastic](https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html) in part **HTTP-HTTP considerations** That way the domains that accept only http work fine, and the full flow is Server B,C,D logs-\>filebeat-\>logstash http out -\> over the internet -\> logstash http in Server A -\> Elasticsearch ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2022, 7:34am UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746/6 "2022-08-16T07:34:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.