Archiving OLD data

Elastic Stack Elasticsearch
1

Can you help me with how to archive data with compression works in elasticsearch.

My cluster have all physical boxes .

I am trying to archive last month indices for querying in case we need it.

2

Have a look at curator project which can help I think to automate that.

3

We didn't buy the licence, Appreciate for any other alternatives.

Thanks.

4

Curator requires no license. It is fully Open Source.

1 Like
6

hi @sujatharudra,

Yes, you can archive data in elasticsearch by using curator.
And by using shrink action you can archive data according to you need.

Shrink action is used to reduce the existing index into new index with few primary shard. Several conditions need to be met in order for index shrinking to take place:
• The index must be marked as read-only
• A (primary or replica) copy of every shard in the index must be relocated to the same node
• The cluster must have health green
• The target index must not exist
• The number of primary shards in the target index must be a factor of the number of primary shards in the source index.
• The number_of_shards of the target index must be specified less than source index's one.
• The index must not contain more than 2,147,483,519 documents in total across all shards that will be shrunk into a single shard on the target index as this is the maximum number of docs that can fit into a single shard.
• The node handling the shrink process must have sufficient free disk space to accommodate a second copy of the existing index.

you can also follow the below shrink action template to archive data.

action: shrink
description: >-
Shrink selected indices on the node with the most available space.
Delete source index after successful shrink, then reroute the shrunk
index with the provided parameters.
options:
ignore_empty_list: True
shrink_node: DETERMINISTIC
node_filters:
permit_masters: False
exclude_nodes: ['not_this_node']
number_of_shards: 1
number_of_replicas: 1
shrink_prefix:
shrink_suffix: '-shrink'
delete_after: True
post_allocation:
allocation_type: include
key: node_tag
value: cold
wait_for_active_shards: 1
extra_settings:
settings:
index.codec: best_compression
wait_for_completion: True
wait_for_rebalance: True
wait_interval: 9
max_wait: -1
filters:

  • filtertype: ...

for more information you can also follow the below link:

I hope the above steps will help you.

Thanks.

7

can you suggest me with out curator

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.