Question on what Elasticsearch considers data vs logs

Elastic Stack Elasticsearch
#1

Hello all,

I just setup Graylog + Elasticsearch for my first time. I'm trying to change the default location that ingested logs are saved. Currently, the default filepath is the OS drive, which will fill up in no time.

Looking at the documentation, I see that there are two filepaths to change in the config file, one for data and one for logs. However, I'm confused by the two. Are logs (/var/logs/elasticsearch) Elasticsearch's own logs, or ingested logs? Or are ingested logs considered data?

In addition to changing the filepath in the .yml file, is there anything else I need to do to make this all work? Anything in Graylog, etc?

Thanks for your time.

#2

Hi @halfbakedprod Welcome to the community and thanks for trying Elasticsearch

1st I love your username :slight_smile:

BTW Great questions here is the quick docs on that

Logs that are ingested into Elasticsearch end up in Indices in Elasticsearch that data is stored in path.data example /var/data/elasticsearch

The Logs that Elasticsearch generates in the course of operating are stored in path.logs example /var/log/elasticsearch

You can set those 2 paths to whichever paths make sense for your deployment.

And of course to complete the equation if you want to ingest the logs generated by elasticsearch back into elasticsearch you would use Filebeat and set the path to harvest the elasticsearch logs from where you pointed the elasticsearch logs to ... i.e. path.logs. We actually have a filebeat module for that.

Typically if you are looking to ingest logs you would use Filebeat or perhaps our new Fleet with the new Elastic Agent which is in Beta