How to retrieve the raw documents

Elastic Stack Elasticsearch
1

Hi All,

I have query with regards to raw documents, for example if we send syslogs, netflow data, firewall logs and windows event logs, we can either send it via logstash or directly elasticsearch. If we send through logstash it parse the data and send to elasticsearch. So all the documents are stored in json format in elasticsearch.

For example if there is any security audits if they want to see the raw data, is it possible to retrieve the raw documents of the data or only the json documents can be retrieved?

if its possible how?
if not what are the other possible ways to achieve it ?

Am not sure if my question is right or wrong?

Please do let me know ,if anyone knows it.

Thanks,
Raj

1 Like
2

Hi Raj,

What happens to the original log files? Its these that any auditors are expecting you to keep as they are the 'original' copies of the data concerned. Sending the contents of these files to Elasticsearch is great for monitoring and searching the contents of these files, but if its a requirement, you should also be keeping the original files somewhere for the required amount of time. Every document in Elasticsearch should contain a field housing of the name of the file the event came from to allow you to pinpoint the original file should you need to find it.

Steve

1 Like
3

Thank you Steve for the info and your input, so does that mean once it’s indexed in to elasticsearch ,we can’t retrieve the raw documents? Is it right ?

Raj

4

You retrieve from Elasticsearch whatever you store in it. So you can ingest log files and keep the original log message in a field.

1 Like
5

As @thiago said, you can store a copy of the raw log entry in elasticsearch and then have a process in place to rebuild a new log file from those entries should you need to, but I know in our case, we have to keep the original files for a time. For example if we need to produce the log files for a criminal investigation, just rebuilding the files from the data stored in elasticsearch might not be sufficient, and we would have to supply the original file.
Elasticsearch itself doesn't do anything with the original log files, its entirely out of scope and nothing to do with Elasticseacrh what files are ingested and what you do with the files afterwards.

1 Like
6

Hi Thiago,

Thanks for the reply .

Iam not getting you completely ,Please let me know the procedure for this. How to keep the original message in a filed and how to retrieve it .

Raj

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.