Connecting filebeat with elasticseach

Elastic Stack Beats
1

I have like 5 elastic nodes in total. One node ingest and coordinating node while two are dedicated master nodes and the other two are dedicated data nodes.

I have elasticsearch install on all my five nodes with respective settings.
What i am confused about and want to know is.
Does i need to include all the elasticsearch nodes ip addresses in filebeat.yml file
which is install on seperate node.
output.elasticsearch.hosts: ['https://x.x.x.x', '' '',"" ]

or i should add only the data nodes. Actually i am really confuse and don't know much about this. Filebeat will be inserting data into elasticsearch so can anyone guide on this.

2

hello, i am trying to do the same thing. dashboards are not working. do i need to install elastic on all my VMs or just in the master one ?

3

you have to install elasticsearch on all vms regardless of what type of node you are trying to have a data, ingest or master.

4

but you can install kibana on seperate vm as well if you want too..... as far as filebeat or logstash is concerned it will be install from where you want to ship data to elasticsearch .... so it can be directly connected to elasticsearch you will not be needed to install filebeat along elasticsearch on that machine.

5

okey thank you very much. this was helpful hope you can find your answer

6

I did lol

7

Hi Daud_Ahmed

Did you ever find out the answer to your original question regarding which hosts must be included in the output.elasticsearch hosts property (master, data, indest)? My guess is it's the data nodes unless you're using logstash or dedicated ingest nodes, but the ingest/pipeline nodes also have to send their data somewhere. I agree it's confusing for a beginner like me. I've searched and searched Elastic's documentation and I just can't find a definite answer. The documentation always just says "Elasticsearch nodes".

8

hi @cporte01,

The thing is they have to be ingest nodes........

Basically, you have to include those nodes which includes in elasticsearch.yml file...

node.ingest: true

if your data nodes have this attribute it means they are acting as both data and ingest node so you will point out to them.

But in case you have node.ingest: false and have dedicated ingest nodes then you will includes only those nodes in

output.elasticsearch.hosts

Hope this help. Feel free to ask if you have any more question.

9

Thank you! I think I finally get it. All Elasticsearch nodes are ingest nodes by default. Our ingest volume is still quite low so until volume increases enough that we need to start using dedicated ingest-only nodes I will send filebeat data to the data nodes. My master-eligible nodes are master-only so they will not be included in the filebeat output.

Thanks again for your help!

10

Great @cporte01 ingest nodes need also depend upon the use of ingest pipeline...

11

best of luck :slight_smile:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.