Delete elasticsearch data/indexes for filebeat and metricbeat - every 30 days

Problem

User wants to Delete Elastic search older 30 days of data / indexes.

Solution

Use kibana to set log / data /index rotation. in the kibana choose from the menu Dev Tools and choose console

First Enable deletion and set limitations for rotate

PUT _cluster/settings
{
“transient”: {
“cluster.routing.allocation.disk.watermark.low”: “85%”,
“cluster.routing.allocation.disk.watermark.high”: “90%”,
“cluster.routing.allocation.disk.watermark.flood_stage”: “95%”,
“cluster.info.update.interval”: “1m”
}
}

set delete option

PUT cluster/_settings
{ “transient”: { “cluster.routing.allocation.disk.threshold_enabled”: false } }

PUT _all/_settings
{“index.blocks.read_only_allow_delete”: null}

Note: the error in the reply sometimes pops when you run the next commands without enabling delete

 Create delete Policy for Filebeat and metricbeat

PUT _ilm/policy/cleanup-history
{
“policy”: {
“phases”: {
“hot”: {
“actions”: {}
},
“delete”: {
“min_age”: “30d”,
“actions”: {
“delete”: {}
}
}
}
}
}

 Assign Policy to exiting indexes

PUT /filebeat*/_settings?pretty
{
“lifecycle.name”: “cleanup-history”
}

PUT /metricbeat*/_settings?pretty
{
“lifecycle.name”: “cleanup-history”
}

Create template for new indexes

PUT /_template/logging_policy_template?pretty
{
“index_patterns”: [“fileabeat*”, “metricbeat*”], “settings”: { “index.lifecycle.name”: “cleanup-history” }

Check disk space

GET /_nodes/_local/stats/fs

or more information

GET /_nodes/stats

To check the new configuration has been set go to ”Stack Management” -> Index Lifecycle Management

Choose from the menu Index Lifecycle Policiesand check that cleanup-history is set

Attachment: commands.txt

Errors you might get when you try to delete without enable the deletion features

{
"error" : {
"root_cause" : [
{
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.16.3-2022.02.18-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [filebeat-7.16.3-2022.02.10-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
}
],
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.16.3-2022.02.18-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [filebeat-7.16.3-2022.02.10-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
},
"status" : 429
}