Export data

Elastic Stack Elasticsearch
1

Hi,

We have a cluster of 7 nodes which is growing up fastly. We want to find a way to "archive" some logs that we must keep (legal inquiries) for several years.

I would like to know if there is a solution / command / way to export old data in a text format?
I want to put those text files somewhere else.

Any other ideas / solution is also welcomed.

Regards

2

Hi,

A frozen index has almost no overhead on the cluster (except for maintaining its metadata in memory) and is read-only. Read-only indices are blocked for write operations, such as docs-index_ or force merges. That could be a solution

3

Hi,

I can close / freeze indices but it will still take much space on data nodes.

I want to free up space but I have to keep some logs about 10 years.. With ILM, Idelete logs after X period of time ( 1 year for example)

Imo, export was one solution to gain space in data nodes.

Regards

4

any news ?

Regards

5

you can use logstash with elasticsearch as input with a your query and as output a file to which you can after archiving them wherever you want. (I imagine that these files can be reindexed).

input {
    elasticsearch {
        hosts => ["host:9200"]
        index => "index-we-are-reading-froml"
        query => '
        {"query": {
        .. 
        #Insert your Elasticsearch query here
                }
            }
            }
        }}'
    }

}
output {
    # see documentation
    file {
        path => ...
        codec => line { format => "custom format: %{message}"}
    }
}

I hope it will help you

6

see https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.