Architecture related question


(Prakash Punj) #1

Folks - Thank you so much for helping newbie like me on ELK. Currently I am evaluating ELK and soon may be going production with it. Have some basic question.

1.) Data ingestion: After initial ingestion of a CSV file to a index in ES, if I have another CSV file with the new data in them, can I ingest new data to the same index in ES. These are all same data data generated on different day.

2.) Currently I m evaluating ELK, so I am running as a user account on a linux machine with ES and Kibana running on the same machine but when we move to production, I would like this to run as a service and not tied to a user account. how can I do that ?

3.) Going forward I may need to add bunch of ES machines, can it be done easily

4.) I noticed that it can also use hadoop for distributed storage (HDFS). Is it a good thing to do to leverage your exising hadoop architrecture

5.) How do use data protection ( authentication/authorization)


(David Pilato) #2

Thank you so much for helping newbie like me on ELK

Read: https://www.elastic.co/elk-stack :wink:

can I ingest new data to the same index in ES?

Yes. As long as you don't use the same _id or let elasticsearch autogenerate it, you can always append new data.

I would like this to run as a service and not tied to a user account. how can I do that ?

Have a look at

And choose the right package for your OS and you should be ok.

Going forward I may need to add bunch of ES machines, can it be done easily

Look at https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#unicast.hosts

But in short: yes

I noticed that it can also use hadoop for distributed storage (HDFS). Is it a good thing to do to leverage your exising hadoop architrecture

So not exactly. You can use HDFS to do snapshot and restore. But don't store your indices on HDFS.

How do use data protection ( authentication/authorization)

X-pack (commercial plugin) provides that.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.