Architecture related question

[Prakash_Punj]

Folks - Thank you so much for helping newbie like me on ELK. Currently I am evaluating ELK and soon may be going production with it. Have some basic question.

1.) Data ingestion: After initial ingestion of a CSV file to a index in ES, if I have another CSV file with the new data in them, can I ingest new data to the same index in ES. These are all same data data generated on different day.

2.) Currently I m evaluating ELK, so I am running as a user account on a linux machine with ES and Kibana running on the same machine but when we move to production, I would like this to run as a service and not tied to a user account. how can I do that ?

3.) Going forward I may need to add bunch of ES machines, can it be done easily

4.) I noticed that it can also use hadoop for distributed storage (HDFS). Is it a good thing to do to leverage your exising hadoop architrecture

5.) How do use data protection ( authentication/authorization)

[dadoonet (David Pilato)]

Thank you so much for helping newbie like me on ELK

Read: Elastic Stack: Elasticsearch, Kibana, Beats & Logstash | Elastic

can I ingest new data to the same index in ES?

Yes. As long as you don't use the same _id or let elasticsearch autogenerate it, you can always append new data.

I would like this to run as a service and not tied to a user account. how can I do that ?

Have a look at

• Installing Elasticsearch | Elasticsearch Guide [8.11] | Elastic
• Install Kibana | Kibana Guide [8.11] | Elastic

And choose the right package for your OS and you should be ok.

Going forward I may need to add bunch of ES machines, can it be done easily

Look at Important Elasticsearch configuration | Elasticsearch Guide [8.11] | Elastic

But in short: yes

I noticed that it can also use hadoop for distributed storage (HDFS). Is it a good thing to do to leverage your exising hadoop architrecture

So not exactly. You can use HDFS to do snapshot and restore. But don't store your indices on HDFS.

How do use data protection ( authentication/authorization)

X-pack (commercial plugin) provides that.