Delete elasticsearch data/indexes for filebeat and metricbeat - every 30 days

Quote from moshe on 17/05/2022, 7:57 pmProblem
User wants to Delete Elastic search older 30 days of data / indexes.
Solution
Use kibana to set log / data /index rotation. in the kibana choose from the menu Dev Tools and choose console
First Enable deletion and set limitations for rotate
PUT _cluster/settings
{
“transient”: {
“cluster.routing.allocation.disk.watermark.low”: “85%”,
“cluster.routing.allocation.disk.watermark.high”: “90%”,
“cluster.routing.allocation.disk.watermark.flood_stage”: “95%”,
“cluster.info.update.interval”: “1m”
}
}set delete option
PUT cluster/_settings
{ “transient”: { “cluster.routing.allocation.disk.threshold_enabled”: false } }PUT _all/_settings
{“index.blocks.read_only_allow_delete”: null}Note: the error in the reply sometimes pops when you run the next commands without enabling delete
Create delete Policy for Filebeat and metricbeat
PUT _ilm/policy/cleanup-history
{
“policy”: {
“phases”: {
“hot”: {
“actions”: {}
},
“delete”: {
“min_age”: “30d”,
“actions”: {
“delete”: {}
}
}
}
}
}Assign Policy to exiting indexes
PUT /filebeat*/_settings?pretty
{
“lifecycle.name”: “cleanup-history”
}PUT /metricbeat*/_settings?pretty
{
“lifecycle.name”: “cleanup-history”
}Create template for new indexes
PUT /_template/logging_policy_template?pretty
{
“index_patterns”: [“fileabeat*”, “metricbeat*”], “settings”: { “index.lifecycle.name”: “cleanup-history” }Check disk space
GET /_nodes/_local/stats/fs
or more information
GET /_nodes/stats
To check the new configuration has been set go to ”Stack Management” -> Index Lifecycle Management
Choose from the menu Index Lifecycle Policiesand check that cleanup-history is set
Problem
User wants to Delete Elastic search older 30 days of data / indexes.
Solution
Use kibana to set log / data /index rotation. in the kibana choose from the menu Dev Tools and choose console
First Enable deletion and set limitations for rotate
PUT _cluster/settings
{
“transient”: {
“cluster.routing.allocation.disk.watermark.low”: “85%”,
“cluster.routing.allocation.disk.watermark.high”: “90%”,
“cluster.routing.allocation.disk.watermark.flood_stage”: “95%”,
“cluster.info.update.interval”: “1m”
}
}
set delete option
PUT cluster/_settings
{ “transient”: { “cluster.routing.allocation.disk.threshold_enabled”: false } }PUT _all/_settings
{“index.blocks.read_only_allow_delete”: null}
Note: the error in the reply sometimes pops when you run the next commands without enabling delete
Create delete Policy for Filebeat and metricbeat
PUT _ilm/policy/cleanup-history
{
“policy”: {
“phases”: {
“hot”: {
“actions”: {}
},
“delete”: {
“min_age”: “30d”,
“actions”: {
“delete”: {}
}
}
}
}
}
Assign Policy to exiting indexes
PUT /filebeat*/_settings?pretty
{
“lifecycle.name”: “cleanup-history”
}PUT /metricbeat*/_settings?pretty
{
“lifecycle.name”: “cleanup-history”
}
Create template for new indexes
PUT /_template/logging_policy_template?pretty
{
“index_patterns”: [“fileabeat*”, “metricbeat*”], “settings”: { “index.lifecycle.name”: “cleanup-history” }
Check disk space
GET /_nodes/_local/stats/fs
or more information
GET /_nodes/stats
To check the new configuration has been set go to ”Stack Management” -> Index Lifecycle Management
Choose from the menu Index Lifecycle Policiesand check that cleanup-history is set
Uploaded files:

Quote from moshe on 17/05/2022, 7:58 pmErrors you might get when you try to delete without enable the deletion features
{
"error" : {
"root_cause" : [
{
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.16.3-2022.02.18-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [filebeat-7.16.3-2022.02.10-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
}
],
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.16.3-2022.02.18-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [filebeat-7.16.3-2022.02.10-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
},
"status" : 429
}
Errors you might get when you try to delete without enable the deletion features
{
"error" : {
"root_cause" : [
{
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.16.3-2022.02.18-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [filebeat-7.16.3-2022.02.10-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
}
],
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.16.3-2022.02.18-000002] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [filebeat-7.16.3-2022.02.10-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
},
"status" : 429
}