How to Get a Raw Logs Copy from elasticsearch

Elastic Stack Elasticsearch
1

Hi Team,

We are currently sending all our audit logs to elasticsearch cluster using filebeat and auditbeat ( Logstash is not used) , Can we get the raw copy of logs or some log forwarder can be used in elasticsearch to get the raw logs to another repository. This is a part of compliance requirements to store the raw logs for a period of time.

Thanks,
Ajesh

2

By default Elasticsearch already stores the raw documents as well as indexing them. You can switch that off by disabling the _source field or manipulating it at index time but assuming you haven't done that you are already storing what you need.

For longer-term storage you can take snapshots (which you can still search) and you can make a backup of the snapshot repository too for archival purposes.

3

Thanks David,

will it be the same case if we use auditbeat and filebeat with ECS?

Also i am not seeing any _source fields in the discovery . How can we see these raw logs?
image

Can we forward these raw logs from elasticsearch to another platform using any connector

4

The source is returned by default in searches, although it's not so useful to see it in the Discovery UI.

Elasticsearch doesn't have a way to push logs to another system, but the other system might be able to pull them from Elasticsearch. I'm not sure why you need another system since Elasticsearch is already retaining the data you want.

5

Thanks David and sorry for the delay

Just wanted to know how we can see/show the raw logs to compliance team. Can you please share me the location of the raw logs or how do we get it using some command from dev console.

Also regarding the pulling of logs , we would like to know if we can integrate this with our Internal tool for analysis like AI/machine learning for the network logs and apache error and access logs etc.

Thanks,
Ajesh

6

Execute a search. For instance GET /auditbeat-*/_search will show you the _source for the first ten documents it finds in indices matching auditbeat-*.

As I said, Elasticsearch won't push logs to another system, but you can have your internal tool pull logs from Elasticsearch.

1 Like
7

Thanks David

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.