Guía Completa de Backup y recuperación a nivel de sistema
El backup y recuperación a nivel de sistema constituye una de las disciplinas más críticas en la gestión de infraestructuras modernas, representando la diferencia entre la continuidad operacional y el desastre empresarial. En un mundo donde los datos son el activo más valioso de las organizaciones y donde el downtime se mide no solo en pérdidas económicas directas sino también en reputación y confianza del cliente, las estrategias de backup y disaster recovery han evolucionado desde simples copias de seguridad hasta ecosistemas complejos de alta disponibilidad y recuperación automatizada.
Esta disciplina ha experimentado una transformación radical con la adopción de tecnologías cloud, contenedores, microservicios y infraestructura distribuida. Los enfoques tradicionales de backup basados en cintas magnéticas y copias programadas han dado paso a estrategias sofisticadas que incluyen replicación en tiempo real, snapshots incrementales, geo-redundancia y automated disaster recovery que pueden restaurar servicios completos en cuestión de minutos.
Esta guía exhaustiva explora todos los aspectos del backup y recuperación a nivel de sistema, desde los conceptos fundamentales hasta implementaciones avanzadas de enterprise, proporcionando las herramientas, estrategias y mejores prácticas necesarias para diseñar y operar sistemas de backup robustos que soporten los requisitos de disponibilidad y recuperación más exigentes de la empresa moderna.
Fundamentos de Backup y Disaster Recovery
Conceptos Esenciales y Terminología
El backup y disaster recovery se fundamenta en conceptos clave que definen los parámetros operacionales y expectativas de servicio. El Recovery Time Objective (RTO) establece el tiempo máximo aceptable para restaurar un servicio después de una interrupción, mientras que el Recovery Point Objective (RPO) define la cantidad máxima de datos que la organización puede permitirse perder medida en tiempo.
Estos conceptos no son simplemente métricas técnicas; representan decisiones de negocio fundamentales que equilibran costo, complejidad y riesgo. Un RTO de minutos requiere arquitecturas de alta disponibilidad con failover automático y redundancia completa, mientras que un RPO de segundos demanda replicación síncrona y consistency controls que pueden impactar significativamente el performance y costo del sistema.
La clasificación de criticidad de sistemas y datos es igualmente importante. Los sistemas tier-1 que soportan funciones de negocio críticas requieren estrategias de backup y recovery diferentes a los sistemas tier-3 de desarrollo o testing. Esta clasificación determina las tecnologías, frecuencias y recursos asignados a cada componente del ecosistema de TI.
Tipos de Backup y Sus Características
Los tipos de backup tradicionales incluyen full backups que copian todos los datos seleccionados, incremental backups que copian solo los cambios desde el último backup de cualquier tipo, y differential backups que copian cambios desde el último full backup. Sin embargo, las tecnologías modernas han expandido significativamente estas opciones.
Los snapshot backups capturan el estado completo del sistema en un punto específico del tiempo, permitiendo restauración instantánea a ese estado. Los continuous data protection (CDP) systems monitoran y capturan cada cambio de datos en tiempo real, proporcionando RPO de segundos o incluso zero data loss.
La replicación puede ser síncrona, donde cada transacción debe ser confirmada en múltiples locations antes de completarse, garantizando zero data loss pero con impacto en performance; o asíncrona, donde las transacciones se confirman localmente y luego se replican, proporcionando mejor performance pero con potencial pérdida de datos durante failures.
#!/bin/bash
## Script de backup multinivel con diferentes estrategias
set -euo pipefail
## Configuración global
BACKUP_BASE_DIR="/backup"
LOG_DIR="/var/log/backup"
RETENTION_FULL=30 # días para backups completos
RETENTION_INCREMENTAL=7 # días para backups incrementales
ENCRYPTION_KEY="/etc/backup/backup.key"
REMOTE_STORAGE="s3://company-backups"
## Logging function
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "${LOG_DIR}/backup.log"
}
## Verificar prerequisites
check_prerequisites() {
log "Verificando prerequisites del sistema de backup..."
# Verificar disponibilidad de herramientas
for tool in rsync gpg aws duplicity; do
if ! command -v "$tool" &> /dev/null; then
log "ERROR: $tool no está instalado"
exit 1
fi
done
# Verificar espacio disponible
available_space=$(df "${BACKUP_BASE_DIR}" | awk 'NR==2 {print $4}')
required_space=1048576 # 1GB en KB
if [ "$available_space" -lt "$required_space" ]; then
log "WARNING: Espacio insuficiente en ${BACKUP_BASE_DIR}"
log "Disponible: ${available_space}KB, Requerido: ${required_space}KB"
fi
# Verificar conectividad remota
if ! aws s3 ls "${REMOTE_STORAGE}/" &> /dev/null; then
log "ERROR: No se puede acceder al almacenamiento remoto ${REMOTE_STORAGE}"
exit 1
fi
log "Prerequisites verificados correctamente"
}
## Backup completo del sistema
full_system_backup() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local backup_dir="${BACKUP_BASE_DIR}/full/${backup_date}"
log "Iniciando backup completo del sistema..."
mkdir -p "${backup_dir}"
# Backup de configuraciones críticas del sistema
log "Respaldando configuraciones del sistema..."
tar -czf "${backup_dir}/system-config.tar.gz" \
/etc \
/var/spool/cron \
/usr/local/etc \
--exclude=/etc/shadow- \
--exclude=/etc/passwd- 2>/dev/null || true
# Backup de datos de aplicaciones
if [ -d "/var/lib/mysql" ]; then
log "Respaldando base de datos MySQL..."
mysqldump --all-databases --single-transaction --routines --triggers \
| gzip > "${backup_dir}/mysql-all-databases.sql.gz"
fi
if [ -d "/var/lib/postgresql" ]; then
log "Respaldando base de datos PostgreSQL..."
sudo -u postgres pg_dumpall | gzip > "${backup_dir}/postgresql-all-databases.sql.gz"
fi
# Backup de home directories
log "Respaldando directorios de usuarios..."
tar -czf "${backup_dir}/home-directories.tar.gz" \
/home \
--exclude=/home/*/tmp \
--exclude=/home/*/cache 2>/dev/null || true
# Backup de logs importantes
log "Respaldando logs del sistema..."
tar -czf "${backup_dir}/system-logs.tar.gz" \
/var/log \
--exclude=/var/log/*.1 \
--exclude=/var/log/*.gz 2>/dev/null || true
# Encriptar backup
log "Encriptando backup completo..."
tar -czf - "${backup_dir}" | \
gpg --cipher-algo AES256 --compress-algo 1 --symmetric \
--passphrase-file "${ENCRYPTION_KEY}" \
--output "${backup_dir}.gpg"
# Subir a almacenamiento remoto
log "Subiendo backup a almacenamiento remoto..."
aws s3 cp "${backup_dir}.gpg" "${REMOTE_STORAGE}/full/" \
--storage-class STANDARD_IA
# Verificar integridad
local local_checksum=$(sha256sum "${backup_dir}.gpg" | cut -d' ' -f1)
aws s3api head-object --bucket $(echo $REMOTE_STORAGE | cut -d'/' -f3) \
--key "full/$(basename ${backup_dir}.gpg)" \
--checksum-mode ENABLED >/dev/null 2>&1
log "Backup completo finalizado: ${backup_dir}.gpg"
log "Checksum: ${local_checksum}"
# Cleanup local
rm -rf "${backup_dir}"
}
## Backup incremental
incremental_backup() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local backup_dir="${BACKUP_BASE_DIR}/incremental/${backup_date}"
local snapshot_file="${BACKUP_BASE_DIR}/.snapshot"
log "Iniciando backup incremental..."
mkdir -p "${backup_dir}"
# Crear snapshot reference si no existe
if [ ! -f "${snapshot_file}" ]; then
find /etc /home /var/lib -type f -newer /dev/null 2>/dev/null > "${snapshot_file}"
fi
# Encontrar archivos modificados desde último backup
find /etc /home /var/lib -type f -newer "${snapshot_file}" 2>/dev/null | \
while IFS= read -r file; do
if [ -f "$file" ]; then
# Crear estructura de directorios
target_dir="${backup_dir}$(dirname "$file")"
mkdir -p "$target_dir"
# Copiar archivo preservando metadatos
cp -p "$file" "$target_dir/"
fi
done
# Actualizar snapshot
touch "${snapshot_file}"
# Comprimir y encriptar
if [ "$(ls -A ${backup_dir})" ]; then
tar -czf "${backup_dir}.tar.gz" -C "${backup_dir}" .
gpg --cipher-algo AES256 --symmetric \
--passphrase-file "${ENCRYPTION_KEY}" \
--output "${backup_dir}.gpg" "${backup_dir}.tar.gz"
# Subir a remoto
aws s3 cp "${backup_dir}.gpg" "${REMOTE_STORAGE}/incremental/"
log "Backup incremental completado: ${backup_dir}.gpg"
# Cleanup
rm -rf "${backup_dir}" "${backup_dir}.tar.gz"
else
log "No hay cambios para backup incremental"
rmdir "${backup_dir}"
fi
}
## Backup de aplicaciones específicas
application_backup() {
local app_name="$1"
local backup_date=$(date +%Y%m%d_%H%M%S)
local backup_dir="${BACKUP_BASE_DIR}/apps/${app_name}/${backup_date}"
mkdir -p "${backup_dir}"
case "$app_name" in
"docker")
log "Respaldando contenedores Docker..."
# Backup de imágenes
docker images --format "table {{.Repository}}:{{.Tag}}" | \
grep -v REPOSITORY | while read image; do
safe_name=$(echo "$image" | tr '/:' '_')
docker save "$image" | gzip > "${backup_dir}/${safe_name}.tar.gz"
done
# Backup de volúmenes
docker volume ls -q | while read volume; do
docker run --rm -v "$volume":/volume -v "${backup_dir}":/backup \
alpine tar czf "/backup/volume_${volume}.tar.gz" -C /volume .
done
# Backup de docker-compose files
find /opt -name "docker-compose.yml" -o -name "docker-compose.yaml" | \
xargs tar czf "${backup_dir}/docker-compose-files.tar.gz" 2>/dev/null || true
;;
"kubernetes")
log "Respaldando recursos de Kubernetes..."
# Backup de recursos del cluster
kubectl get all --all-namespaces -o yaml > "${backup_dir}/all-resources.yaml"
kubectl get configmaps --all-namespaces -o yaml > "${backup_dir}/configmaps.yaml"
kubectl get secrets --all-namespaces -o yaml > "${backup_dir}/secrets.yaml"
kubectl get pv -o yaml > "${backup_dir}/persistent-volumes.yaml"
kubectl get pvc --all-namespaces -o yaml > "${backup_dir}/persistent-volume-claims.yaml"
# Backup de ETCD si es accesible
if kubectl get pods -n kube-system | grep etcd; then
kubectl exec -n kube-system etcd-$(hostname) -- \
etcdctl snapshot save /tmp/etcd-snapshot.db
kubectl cp kube-system/etcd-$(hostname):/tmp/etcd-snapshot.db \
"${backup_dir}/etcd-snapshot.db"
fi
;;
"gitlab")
log "Respaldando GitLab..."
if command -v gitlab-backup &> /dev/null; then
gitlab-backup create BACKUP=backup_${backup_date}
cp /var/opt/gitlab/backups/backup_${backup_date}_gitlab_backup.tar \
"${backup_dir}/"
# Backup de configuración
cp -r /etc/gitlab "${backup_dir}/gitlab-config"
fi
;;
esac
# Comprimir, encriptar y subir
if [ "$(ls -A ${backup_dir})" ]; then
tar -czf "${backup_dir}.tar.gz" -C "$(dirname ${backup_dir})" "$(basename ${backup_dir})"
gpg --cipher-algo AES256 --symmetric \
--passphrase-file "${ENCRYPTION_KEY}" \
--output "${backup_dir}.gpg" "${backup_dir}.tar.gz"
aws s3 cp "${backup_dir}.gpg" "${REMOTE_STORAGE}/apps/${app_name}/"
log "Backup de ${app_name} completado"
# Cleanup
rm -rf "${backup_dir}" "${backup_dir}.tar.gz"
fi
}
## Cleanup de backups antiguos
cleanup_old_backups() {
log "Limpiando backups antiguos..."
# Cleanup local
find "${BACKUP_BASE_DIR}/full" -name "*.gpg" -mtime +${RETENTION_FULL} -delete 2>/dev/null || true
find "${BACKUP_BASE_DIR}/incremental" -name "*.gpg" -mtime +${RETENTION_INCREMENTAL} -delete 2>/dev/null || true
# Cleanup remoto
aws s3 ls "${REMOTE_STORAGE}/full/" --recursive | \
awk '$1 <= "'$(date -d "${RETENTION_FULL} days ago" '+%Y-%m-%d')'" {print $4}' | \
while read file; do
aws s3 rm "${REMOTE_STORAGE}/${file}"
done
aws s3 ls "${REMOTE_STORAGE}/incremental/" --recursive | \
awk '$1 <= "'$(date -d "${RETENTION_INCREMENTAL} days ago" '+%Y-%m-%d')'" {print $4}' | \
while read file; do
aws s3 rm "${REMOTE_STORAGE}/${file}"
done
log "Cleanup de backups completado"
}
## Función principal
main() {
local backup_type="${1:-full}"
mkdir -p "${LOG_DIR}" "${BACKUP_BASE_DIR}"/{full,incremental,apps}
check_prerequisites
case "$backup_type" in
"full")
full_system_backup
;;
"incremental")
incremental_backup
;;
"app")
if [ -z "${2:-}" ]; then
log "ERROR: Especifica el nombre de la aplicación"
exit 1
fi
application_backup "$2"
;;
"cleanup")
cleanup_old_backups
;;
*)
echo "Uso: $0 {full|incremental|app <name>|cleanup}"
exit 1
;;
esac
log "Operación de backup completada exitosamente"
}
## Ejecutar función principal
main "$@"Estrategias de Backup para Diferentes Tecnologías
Backup de Bases de Datos a Escala
Las bases de datos representan uno de los componentes más críticos para backup y recovery, requiriendo estrategias especializadas que consideren consistency, performance impact, y recovery time requirements. Las técnicas varían significativamente según el tipo de base de datos y arquitectura de deployment.
-- Script de backup automatizado para PostgreSQL con replicación
-- /opt/backup/scripts/postgresql-backup.sql
-- Configuración de backup para PostgreSQL con replicación streaming
-- Este script debe ejecutarse en el servidor primary
BEGIN;
-- Crear función para backup consistente con WAL
CREATE OR REPLACE FUNCTION perform_base_backup(backup_label text)
RETURNS TABLE(checkpoint_lsn text, backup_file text) AS $$
DECLARE
start_lsn text;
backup_path text;
wal_files text[];
BEGIN
-- Iniciar backup base
SELECT pg_start_backup(backup_label, false, false) INTO start_lsn;
-- Determinar ruta de backup
backup_path := '/backup/postgresql/' || to_char(now(), 'YYYY-MM-DD_HH24-MI-SS');
-- Crear directorio de backup
PERFORM pg_file_write(backup_path || '/backup_label',
backup_label || E'\nSTART WAL LOCATION: ' || start_lsn, false);
-- Realizar copia física de datos
-- Nota: Esta parte requiere implementación externa usando pg_basebackup
-- Finalizar backup y obtener WAL final
SELECT pg_stop_backup(false, true) INTO checkpoint_lsn;
RETURN QUERY SELECT checkpoint_lsn, backup_path;
END;
$$ LANGUAGE plpgsql;
-- Función para backup incremental usando WAL shipping
CREATE OR REPLACE FUNCTION setup_wal_archiving()
RETURNS boolean AS $$
BEGIN
-- Configurar archivado de WAL
-- Estas configuraciones normalmente van en postgresql.conf
-- archive_mode = on
-- archive_command = 'rsync %p /backup/postgresql/wal/%f'
-- wal_level = replica
-- max_wal_senders = 3
-- wal_keep_segments = 32
-- Verificar configuración actual
IF current_setting('archive_mode') != 'on' THEN
RAISE WARNING 'WAL archiving no está habilitado. Configurar archive_mode = on';
RETURN false;
END IF;
IF current_setting('wal_level') NOT IN ('replica', 'logical') THEN
RAISE WARNING 'wal_level debe ser replica o logical para backup streaming';
RETURN false;
END IF;
RETURN true;
END;
$$ LANGUAGE plpgsql;
-- Función para backup de esquemas y datos específicos
CREATE OR REPLACE FUNCTION backup_schema_data(schema_name text, output_file text)
RETURNS boolean AS $$
DECLARE
cmd text;
result integer;
BEGIN
-- Crear comando pg_dump para esquema específico
cmd := format('pg_dump --schema=%L --data-only --inserts --column-inserts --no-owner --no-privileges -f %L',
schema_name, output_file);
-- Ejecutar backup (requiere extensión plsh o similar)
-- En implementación real, esto se haría desde script externo
RETURN true;
END;
$$ LANGUAGE plpgsql;
COMMIT;
bash
#!/bin/bash
## Script de backup completo para MongoDB replica set
set -euo pipefail
## Configuración
MONGO_HOST="mongodb://mongo-primary:27017,mongo-secondary1:27017,mongo-secondary2:27017"
MONGO_REPLICA_SET="rs0"
BACKUP_DIR="/backup/mongodb"
RETENTION_DAYS=7
S3_BUCKET="s3://company-backups/mongodb"
## Función de logging
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a /var/log/mongodb-backup.log
}
## Backup usando mongodump con oplock replay
mongodb_consistent_backup() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local backup_path="${BACKUP_DIR}/${backup_date}"
log "Iniciando backup consistente de MongoDB..."
mkdir -p "$backup_path"
# Obtener información del replica set
mongo --host "$MONGO_HOST" --eval "
rs.status().members.forEach(function(member) {
print('Member: ' + member.name + ' State: ' + member.stateStr);
});
" --quiet
# Realizar backup con oplog para consistencia
mongodump \
--host "$MONGO_HOST" \
--readPreference secondary \
--oplog \
--out "$backup_path" \
--gzip
# Backup de configuración del cluster
mongo --host "$MONGO_HOST" --eval "
db.runCommand('listCollections').cursor.firstBatch.forEach(
function(collection) {
printjson(collection);
}
);
" --quiet > "${backup_path}/collections_info.json"
# Backup de usuarios y roles
mongoexport \
--host "$MONGO_HOST" \
--db admin \
--collection system.users \
--out "${backup_path}/users.json"
mongoexport \
--host "$MONGO_HOST" \
--db admin \
--collection system.roles \
--out "${backup_path}/roles.json"
# Comprimir backup
tar -czf "${backup_path}.tar.gz" -C "$BACKUP_DIR" "$(basename $backup_path)"
# Subir a S3 con metadata
aws s3 cp "${backup_path}.tar.gz" "$S3_BUCKET/" \
--metadata "backup-date=${backup_date},replica-set=${MONGO_REPLICA_SET}" \
--storage-class STANDARD_IA
# Verificar backup
if mongorestore --host "$MONGO_HOST" --dry-run --dir "$backup_path" &>/dev/null; then
log "Backup verificado exitosamente: ${backup_path}.tar.gz"
else
log "ERROR: Backup verification failed"
return 1
fi
# Cleanup local
rm -rf "$backup_path"
return 0
}
## Backup punto-en-tiempo usando oplogs
mongodb_pit_backup() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local oplog_backup_path="${BACKUP_DIR}/oplog_${backup_date}"
log "Realizando backup de oplog para point-in-time recovery..."
mkdir -p "$oplog_backup_path"
# Obtener timestamp actual
local current_timestamp=$(mongo --host "$MONGO_HOST" --eval "
db.runCommand('isMaster').operationTime.toString()
" --quiet)
# Backup de oplog desde última sincronización
mongodump \
--host "$MONGO_HOST" \
--db local \
--collection oplog.rs \
--query "{ts: {\$gte: Timestamp($(date -d '1 hour ago' +%s), 0)}}" \
--out "$oplog_backup_path"
# Comprimir y subir
tar -czf "${oplog_backup_path}.tar.gz" -C "$BACKUP_DIR" "$(basename $oplog_backup_path)"
aws s3 cp "${oplog_backup_path}.tar.gz" "$S3_BUCKET/oplog/" \
--metadata "timestamp=${current_timestamp}"
# Cleanup
rm -rf "$oplog_backup_path"
log "Oplog backup completado: ${oplog_backup_path}.tar.gz"
}
## Ejecutar backups
mongodb_consistent_backup
mongodb_pit_backup
## Cleanup de backups antiguos
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete
aws s3 ls "$S3_BUCKET/" | awk '$1 <= "'$(date -d "$RETENTION_DAYS days ago" '+%Y-%m-%d')'" {print $4}' | \
while read file; do aws s3 rm "$S3_BUCKET/$file"; done
log "MongoDB backup process completed"Backup de Aplicaciones Cloud-Native
Las aplicaciones cloud-native requieren estrategias de backup que consideren la naturaleza distribuida, ephemeral storage, y service dependencies. Esto incluye backup de container images, persistent volumes, configuration data, y service definitions.
## Kubernetes backup automation con Velero
apiVersion: v1
kind: ConfigMap
metadata:
name: velero-backup-config
namespace: velero
data:
backup-schedule.yaml: |
# Backup diario completo del cluster
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: daily-cluster-backup
namespace: velero
spec:
schedule: "0 2 * * *" # 2 AM daily
template:
metadata:
name: daily-cluster-backup-{{.CreationTimestamp}}
spec:
includedNamespaces:
- production
- staging
- monitoring
excludedNamespaces:
- kube-system
- kube-public
includedResources:
- "*"
excludedResources:
- events
- events.events.k8s.io
storageLocation: aws-s3-backup
volumeSnapshotLocations:
- aws-ebs-snapshots
ttl: 720h0m0s # 30 days retention
snapshotVolumes: true
includeClusterResources: true
hooks:
resources:
- name: database-backup-hook
includedNamespaces:
- production
labelSelector:
matchLabels:
app: postgresql
pre:
- exec:
command:
- /bin/bash
- -c
- "pg_dump -h localhost -U postgres -d myapp > /tmp/db-backup.sql"
container: postgresql
post:
- exec:
command:
- /bin/bash
- -c
- "rm -f /tmp/db-backup.sql"
container: postgresql
---
# Backup incremental cada 4 horas
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: incremental-backup
namespace: velero
spec:
schedule: "0 */4 * * *"
template:
metadata:
name: incremental-backup-{{.CreationTimestamp}}
spec:
includedNamespaces:
- production
includedResources:
- configmaps
- secrets
- persistentvolumeclaims
- persistentvolumes
storageLocation: aws-s3-backup
ttl: 168h0m0s # 7 days retention
storage-location.yaml: |
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
name: aws-s3-backup
namespace: velero
spec:
provider: aws
objectStorage:
bucket: company-kubernetes-backups
prefix: velero
config:
region: us-east-1
s3ForcePathStyle: "false"
serverSideEncryption: AES256
volume-snapshot-location.yaml: |
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
name: aws-ebs-snapshots
namespace: velero
spec:
provider: aws
config:
region: us-east-1
---
## Custom Resource para backup selectivo de aplicaciones
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: appbackups.backup.company.com
spec:
group: backup.company.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
appName:
type: string
namespaces:
type: array
items:
type: string
schedule:
type: string
retentionDays:
type: integer
minimum: 1
maximum: 365
backupHooks:
type: object
properties:
pre:
type: array
items:
type: object
properties:
name:
type: string
command:
type: array
items:
type: string
post:
type: array
items:
type: object
properties:
name:
type: string
command:
type: array
items:
type: string
storageClass:
type: string
encryption:
type: object
properties:
enabled:
type: boolean
keyId:
type: string
status:
type: object
properties:
lastBackupTime:
type: string
lastBackupStatus:
type: string
nextScheduledBackup:
type: string
scope: Namespaced
names:
plural: appbackups
singular: appbackup
kind: AppBackup
---
## Ejemplo de uso del CRD para backup de aplicación específica
apiVersion: backup.company.com/v1
kind: AppBackup
metadata:
name: ecommerce-app-backup
namespace: production
spec:
appName: ecommerce
namespaces:
- ecommerce-frontend
- ecommerce-backend
- ecommerce-database
schedule: "0 1 * * *" # 1 AM daily
retentionDays: 30
backupHooks:
pre:
- name: stop-background-jobs
command:
- /bin/sh
- -c
- "kubectl scale deployment background-worker --replicas=0"
- name: flush-cache
command:
- /bin/sh
- -c
- "redis-cli -h cache-redis FLUSHALL"
post:
- name: restart-background-jobs
command:
- /bin/sh
- -c
- "kubectl scale deployment background-worker --replicas=3"
storageClass: fast-ssd
encryption:
enabled: true
keyId: "arn:aws:kms:us-east-1:123456789:key/12345678-1234-1234-1234-123456789012"Backup de Infrastructure as
El backup de Infrastructure as Code (IaC) incluye no solo los archivos de configuración sino también el estado, versioning, y dependencies. Esto es crucial para disaster recovery y audit compliance.
#!/bin/bash
## Backup completo de Infrastructure as
set -euo pipefail
## Configuración
TERRAFORM_DIR="/infrastructure/terraform"
ANSIBLE_DIR="/infrastructure/ansible"
KUBERNETES_MANIFESTS="/infrastructure/k8s"
BACKUP_DIR="/backup/infrastructure"
GIT_REPOS=(
"[email protected]:company/terraform-infrastructure.git"
"[email protected]:company/ansible-playbooks.git"
"[email protected]:company/kubernetes-manifests.git"
"[email protected]:company/helm-charts.git"
)
S3_BUCKET="s3://company-infrastructure-backups"
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a /var/log/iac-backup.log
}
## Backup de repositorios Git con historia completa
backup_git_repositories() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local git_backup_dir="${BACKUP_DIR}/git/${backup_date}"
mkdir -p "$git_backup_dir"
for repo in "${GIT_REPOS[@]}"; do
local repo_name=$(basename "$repo" .git)
log "Backing up Git repository: $repo_name"
# Clone bare repository (incluye toda la historia)
git clone --bare "$repo" "${git_backup_dir}/${repo_name}.git"
# Backup de configuración específica del repo
if [ -d "${git_backup_dir}/${repo_name}.git/hooks" ]; then
tar -czf "${git_backup_dir}/${repo_name}-hooks.tar.gz" \
-C "${git_backup_dir}/${repo_name}.git" hooks/
fi
# Export de metadata del repositorio
cd "${git_backup_dir}/${repo_name}.git"
git log --oneline --all > "../${repo_name}-commits.log"
git branch -a > "../${repo_name}-branches.log"
git tag -l > "../${repo_name}-tags.log"
cd - > /dev/null
done
# Comprimir backup de repositorio Git
tar -czf "${git_backup_dir}.tar.gz" -C "${BACKUP_DIR}/git" "$(basename $git_backup_dir)"
# Upload a S3
aws s3 cp "${git_backup_dir}.tar.gz" "$S3_BUCKET/git/" \
--metadata "backup-type=git,date=${backup_date}"
# Cleanup local
rm -rf "$git_backup_dir"
log "Git repositories backup completed"
}
## Backup de estado de Terraform
backup_terraform_state() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local tf_backup_dir="${BACKUP_DIR}/terraform/${backup_date}"
mkdir -p "$tf_backup_dir"
log "Backing up Terraform state files..."
# Encontrar todos los archivos de estado
find "$TERRAFORM_DIR" -name "*.tfstate*" -type f | while read statefile; do
local relative_path=$(realpath --relative-to="$TERRAFORM_DIR" "$statefile")
local backup_path="${tf_backup_dir}/${relative_path}"
mkdir -p "$(dirname "$backup_path")"
cp "$statefile" "$backup_path"
done
# Backup de archivos de configuración de Terraform
find "$TERRAFORM_DIR" -name "*.tf" -o -name "*.tfvars" | while read tffile; do
local relative_path=$(realpath --relative-to="$TERRAFORM_DIR" "$tffile")
local backup_path="${tf_backup_dir}/config/${relative_path}"
mkdir -p "$(dirname "$backup_path")"
cp "$tffile" "$backup_path"
done
# Backup de providers y módulos lock file
find "$TERRAFORM_DIR" -name ".terraform.lock.hcl" | while read lockfile; do
local relative_path=$(realpath --relative-to="$TERRAFORM_DIR" "$lockfile")
local backup_path="${tf_backup_dir}/locks/${relative_path}"
mkdir -p "$(dirname "$backup_path")"
cp "$lockfile" "$backup_path"
done
# Exportar plan actual si es posible
find "$TERRAFORM_DIR" -name "*.tf" -type f -exec dirname {} \; | sort -u | while read dir; do
if [ -f "$dir/main.tf" ]; then
cd "$dir"
terraform plan -out="${tf_backup_dir}/plans/$(basename $dir).tfplan" 2>/dev/null || {
log "Warning: Could not create plan for $dir"
}
cd - > /dev/null
fi
done
# Comprimir y subir
tar -czf "${tf_backup_dir}.tar.gz" -C "${BACKUP_DIR}/terraform" "$(basename $tf_backup_dir)"
aws s3 cp "${tf_backup_dir}.tar.gz" "$S3_BUCKET/terraform/" \
--metadata "backup-type=terraform,date=${backup_date}"
# Cleanup
rm -rf "$tf_backup_dir"
log "Terraform backup completed"
}
## Backup de inventarios y playbooks de Ansible
backup_ansible_inventory() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local ansible_backup_dir="${BACKUP_DIR}/ansible/${backup_date}"
mkdir -p "$ansible_backup_dir"
log "Backing up Ansible configuration..."
# Backup de inventarios
cp -r "$ANSIBLE_DIR/inventories" "$ansible_backup_dir/" 2>/dev/null || true
# Backup de playbooks
cp -r "$ANSIBLE_DIR/playbooks" "$ansible_backup_dir/" 2>/dev/null || true
# Backup de roles
cp -r "$ANSIBLE_DIR/roles" "$ansible_backup_dir/" 2>/dev/null || true
# Backup de configuración
cp "$ANSIBLE_DIR/ansible.cfg" "$ansible_backup_dir/" 2>/dev/null || true
# Backup de archivos encriptados con Vault (metadata únicamente)
find "$ANSIBLE_DIR" -name "*.yml" -o -name "*.yaml" | \
xargs grep -l "^\$ANSIBLE_VAULT" | while read vaultfile; do
echo "Encrypted file: $vaultfile" >> "${ansible_backup_dir}/vault-files.log"
# No incluir contenido por seguridad, solo metadatos
stat "$vaultfile" >> "${ansible_backup_dir}/vault-files.log"
done
# Export de configuración de host facts si están cacheados
if [ -d "/tmp/ansible_facts_cache" ]; then
cp -r /tmp/ansible_facts_cache "$ansible_backup_dir/" 2>/dev/null || true
fi
# Comprimir y subir
tar -czf "${ansible_backup_dir}.tar.gz" -C "${BACKUP_DIR}/ansible" "$(basename $ansible_backup_dir)"
aws s3 cp "${ansible_backup_dir}.tar.gz" "$S3_BUCKET/ansible/" \
--metadata "backup-type=ansible,date=${backup_date}"
# Cleanup
rm -rf "$ansible_backup_dir"
log "Ansible backup completed"
}
## Backup de manifiestos de Kubernetes y configuración
backup_kubernetes_config() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local k8s_backup_dir="${BACKUP_DIR}/kubernetes/${backup_date}"
mkdir -p "$k8s_backup_dir"
log "Backing up Kubernetes configuration..."
# Backup de manifiestos YAML
if [ -d "$KUBERNETES_MANIFESTS" ]; then
cp -r "$KUBERNETES_MANIFESTS" "$k8s_backup_dir/manifests/"
fi
# Export de configuración actual del cluster
kubectl cluster-info dump --all-namespaces \
--output-directory="${k8s_backup_dir}/cluster-dump" 2>/dev/null || {
log "Warning: Could not create cluster dump"
}
# Export de recursos críticos
for resource in namespaces configmaps secrets services deployments statefulsets daemonsets; do
kubectl get "$resource" --all-namespaces -o yaml > \
"${k8s_backup_dir}/${resource}.yaml" 2>/dev/null || true
done
# Backup de RBAC
kubectl get clusterroles,clusterrolebindings,roles,rolebindings \
--all-namespaces -o yaml > "${k8s_backup_dir}/rbac.yaml" 2>/dev/null || true
# Backup de custom resources
kubectl api-resources --verbs=list --namespaced -o name | \
while read resource; do
kubectl get "$resource" --all-namespaces -o yaml > \
"${k8s_backup_dir}/custom-${resource//\//-}.yaml" 2>/dev/null || true
done
# Comprimir y subir
tar -czf "${k8s_backup_dir}.tar.gz" -C "${BACKUP_DIR}/kubernetes" "$(basename $k8s_backup_dir)"
aws s3 cp "${k8s_backup_dir}.tar.gz" "$S3_BUCKET/kubernetes/" \
--metadata "backup-type=kubernetes,date=${backup_date}"
# Cleanup
rm -rf "$k8s_backup_dir"
log "Kubernetes backup completed"
}
## Crear manifesto de backup con metadata
create_backup_manifest() {
local backup_date=$(date +%Y%m%d_%H%M%S)
local manifest_file="${BACKUP_DIR}/manifest_${backup_date}.json"
cat > "$manifest_file" <EOF
{
"backup_timestamp": "$(date -Iseconds)",
"backup_type": "infrastructure_as_code",
"components": {
"git_repositories": $(printf '%s\n' "${GIT_REPOS[@]}" | jq -R . | jq -s .),
"terraform_directories": ["$TERRAFORM_DIR"],
"ansible_directories": ["$ANSIBLE_DIR"],
"kubernetes_manifests": ["$KUBERNETES_MANIFESTS"]
},
"environment": {
"terraform_version": "$(terraform version -json 2>/dev/null | jq -r .terraform_version || echo 'unknown')",
"ansible_version": "$(ansible --version 2>/dev/null | head -1 || echo 'unknown')",
"kubectl_version": "$(kubectl version --client -o json 2>/dev/null | jq -r .clientVersion.gitVersion || echo 'unknown')"
},
"backup_size_mb": $(du -sm "$BACKUP_DIR" | cut -f1),
"retention_policy": "30 days",
"encryption": "AES256",
"compression": "gzip"
}
EOF
# Subir manifest
aws s3 cp "$manifest_file" "$S3_BUCKET/manifests/" \
--metadata "backup-date=${backup_date}"
log "Backup manifest created: $manifest_file"
}
## Cleanup de backups antiguos
cleanup_old_backups() {
log "Cleaning up old backups..."
local retention_days=30
# Cleanup local
find "$BACKUP_DIR" -type f -name "*.tar.gz" -mtime +$retention_days -delete
# Cleanup S3
for prefix in git terraform ansible kubernetes manifests; do
aws s3api list-objects-v2 --bucket $(echo $S3_BUCKET | cut -d'/' -f3) \
--prefix "$prefix/" --query "Contents[?LastModified<'$(date -d "$retention_days days ago" --iso-8601)'].Key" \
--output text | while read key; do
if [ -n "$key" ] && [ "$key" != "None" ]; then
aws s3 rm "$S3_BUCKET/$key"
fi
done
done
log "Cleanup completed"
}
## Función principal
main() {
mkdir -p "$BACKUP_DIR"/{git,terraform,ansible,kubernetes}
log "Starting Infrastructure as Code backup..."
backup_git_repositories
backup_terraform_state
backup_ansible_inventory
backup_kubernetes_config
create_backup_manifest
cleanup_old_backups
log "Infrastructure as Code backup completed successfully"
}
## Ejecutar backup
mainImplementación de Disaster Recovery
Esta implementación requiere atención a los detalles y seguimiento de las mejores prácticas.
Automated Failover y High Availability
El disaster recovery moderno va más allá del backup tradicional, implementando sistemas de alta disponibilidad que pueden automáticamente detectar failures y realizar failover sin intervención humana. Esto incluye database replication, load balancer health checks, y automated DNS failover.
#!/usr/bin/env python3
"""
Sistema automatizado de disaster recovery y failover
Monitorea servicios críticos y ejecuta procedimientos de recovery automático
"""
import asyncio
import json
import logging
import time
from dataclasses import dataclass, field
from datetime import datetime, timedelta
from enum import Enum
from typing import Dict, List, Optional, Any # Tipos para type hints
import aiohttp
import asyncpg
import boto3
from prometheus_client.parser import text_string_to_metric_families
## Configuración de logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('/var/log/disaster-recovery.log'),
logging.StreamHandler()
]
)
logger = logging.getLogger(__name__)
class ServiceStatus(Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
FAILED = "failed"
RECOVERING = "recovering"
UNKNOWN = "unknown"
class FailoverStrategy(Enum):
ACTIVE_PASSIVE = "active_passive"
ACTIVE_ACTIVE = "active_active"
BLUE_GREEN = "blue_green"
CANARY = "canary"
@dataclass
class HealthCheck:
name: str
url: str
timeout: int = 30
expected_status: int = 200
critical: bool = True
check_interval: int = 30
failure_threshold: int = 3
recovery_threshold: int = 2
@dataclass
class ServiceEndpoint:
name: str
primary_url: str
secondary_url: Optional[str] = None
health_checks: List[HealthCheck] = field(default_factory=list)
failover_strategy: FailoverStrategy = FailoverStrategy.ACTIVE_PASSIVE
auto_failback: bool = False
failback_delay: int = 300 # seconds
@dataclass
class FailoverEvent:
timestamp: datetime
service: str
source: str
target: str
reason: str
success: bool
recovery_time: Optional[float] = None
class DisasterRecoveryOrchestrator:
def __init__(self, config_file: str = '/etc/disaster-recovery/config.json'):
self.config = self._load_config(config_file)
self.services: Dict[str, ServiceEndpoint] = {}
self.service_states: Dict[str, ServiceStatus] = {}
self.failure_counts: Dict[str, int] = {}
self.recovery_counts: Dict[str, int] = {}
self.last_failover: Dict[str, datetime] = {}
self.failover_history: List[FailoverEvent] = []
# AWS clients para DNS failover
self.route53_client = boto3.client('route53',
region_name=self.config.get('aws', {}).get('region', 'us-east-1'))
# Database connections pool
self.db_pool = None
def _load_config(self, config_file: str) -> Dict[str, Any]:
"""Cargar configuración desde archivo JSON"""
try:
with open(config_file, 'r') as f:
return json.load(f)
except FileNotFoundError:
logger.warning(f"Config file {config_file} not found, using defaults")
return self._default_config()
def _default_config(self) -> Dict[str, Any]:
"""Configuración por defecto"""
return {
"services": {
"web_app": {
"primary_url": "https://app.company.com",
"secondary_url": "https://app-backup.company.com",
"health_checks": [
{
"name": "main_health",
"url": "/health",
"timeout": 10,
"expected_status": 200
},
{
"name": "api_health",
"url": "/api/health",
"timeout": 15,
"expected_status": 200
}
],
"failover_strategy": "active_passive",
"auto_failback": True,
"failback_delay": 300
},
"database": {
"primary_url": "postgres://db-primary.company.com:5432",
"secondary_url": "postgres://db-replica.company.com:5432",
"health_checks": [
{
"name": "db_connection",
"url": "postgres://healthcheck",
"timeout": 5,
"critical": True
}
],
"failover_strategy": "active_passive",
"auto_failback": False
}
},
"notification": {
"slack_webhook": "https://hooks.slack.com/services/...",
"email_smtp": {
"server": "smtp.company.com",
"port": 587,
"username": "[email protected]"
}
},
"aws": {
"region": "us-east-1",
"route53_zone_id": "Z1234567890ABC"
}
}
async def initialize(self):
"""Inicializar el sistema de disaster recovery"""
logger.info("Initializing Disaster Recovery Orchestrator...")
# Cargar servicios desde configuración
for service_name, service_config in self.config['services'].items():
health_checks = [
HealthCheck(
name=hc['name'],
url=hc['url'],
timeout=hc.get('timeout', 30),
expected_status=hc.get('expected_status', 200),
critical=hc.get('critical', True),
check_interval=hc.get('check_interval', 30),
failure_threshold=hc.get('failure_threshold', 3),
recovery_threshold=hc.get('recovery_threshold', 2)
) for hc in service_config.get('health_checks', [])
]
self.services[service_name] = ServiceEndpoint(
name=service_name,
primary_url=service_config['primary_url'],
secondary_url=service_config.get('secondary_url'),
health_checks=health_checks,
failover_strategy=FailoverStrategy(
service_config.get('failover_strategy', 'active_passive')
),
auto_failback=service_config.get('auto_failback', False),
failback_delay=service_config.get('failback_delay', 300)
)
# Inicializar estado
self.service_states[service_name] = ServiceStatus.UNKNOWN
self.failure_counts[service_name] = 0
self.recovery_counts[service_name] = 0
# Inicializar pool de conexiones de base de datos
if 'database' in self.services:
try:
db_config = self.config['services']['database']
self.db_pool = await asyncpg.create_pool(
db_config['primary_url'],
min_size=2,
max_size=10,
command_timeout=5
)
except Exception as e:
logger.error(f"Failed to initialize database pool: {e}")
logger.info(f"Initialized monitoring for {len(self.services)} services")
async def check_service_health(self, service: ServiceEndpoint) -> ServiceStatus:
"""Verificar salud de un servicio específico"""
healthy_checks = 0
critical_failures = 0
async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=30)) as session:
for health_check in service.health_checks:
try:
check_url = f"{service.primary_url.rstrip('/')}{health_check.url}"
async with session.get(
check_url,
timeout=aiohttp.ClientTimeout(total=health_check.timeout)
) as response:
if response.status == health_check.expected_status:
healthy_checks += 1
elif health_check.critical:
critical_failures += 1
logger.warning(
f"Critical health check failed for {service.name}: "
f"{health_check.name} returned {response.status}"
)
except asyncio.TimeoutError:
if health_check.critical:
critical_failures += 1
logger.warning(f"Health check timeout for {service.name}: {health_check.name}")
except Exception as e:
if health_check.critical:
critical_failures += 1
logger.error(f"Health check error for {service.name}: {e}")
# Determinar estado basado en resultados
if critical_failures > 0:
return ServiceStatus.FAILED
elif healthy_checks == len(service.health_checks):
return ServiceStatus.HEALTHY
elif healthy_checks > 0:
return ServiceStatus.DEGRADED
else:
return ServiceStatus.FAILED
async def check_database_health(self) -> ServiceStatus:
"""Verificación específica de salud de base de datos"""
if not self.db_pool:
return ServiceStatus.UNKNOWN
try:
async with self.db_pool.acquire() as conn:
# Test simple query
result = await conn.fetchval("SELECT 1")
if result == 1:
# Test más complejo - verificar replication lag
lag_query = """
SELECT CASE
WHEN pg_is_in_recovery() THEN
EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))::int
ELSE 0
END as lag_seconds
"""
lag = await conn.fetchval(lag_query)
if lag is not None and lag 30: # Less than 30 seconds lag
return ServiceStatus.HEALTHY
elif lag is not None and lag 300: # Less than 5 minutes lag
return ServiceStatus.DEGRADED
else:
return ServiceStatus.FAILED
else:
return ServiceStatus.FAILED
except Exception as e:
logger.error(f"Database health check failed: {e}")
return ServiceStatus.FAILED
async def perform_dns_failover(self, service_name: str, target_url: str) -> bool:
"""Realizar failover a nivel DNS usando Route53"""
try:
zone_id = self.config['aws']['route53_zone_id']
# Obtener record actual
response = self.route53_client.list_resource_record_sets(
HostedZoneId=zone_id,
StartRecordName=service_name,
StartRecordType='A'
)
# Actualizar record para apuntar al target
change_batch = {
'Changes': [{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': f"{service_name}.company.com",
'Type': 'A',
'TTL': 60, # Low TTL for fast failover
'ResourceRecords': [
{'Value': self._extract_ip_from_url(target_url)}
]
}
}]
}
change_response = self.route53_client.change_resource_record_sets(
HostedZoneId=zone_id,
ChangeBatch=change_batch
)
# Wait for change to propagate
change_id = change_response['ChangeInfo']['Id']
waiter = self.route53_client.get_waiter('resource_record_sets_changed')
await asyncio.get_event_loop().run_in_executor(
None, waiter.wait, {'Id': change_id}
)
logger.info(f"DNS failover completed for {service_name} to {target_url}")
return True
except Exception as e:
logger.error(f"DNS failover failed for {service_name}: {e}")
return False
def _extract_ip_from_url(self, url: str) -> str:
"""Extraer IP de URL (simplified - en producción usar DNS lookup)"""
import socket
from urllib.parse import urlparse
parsed = urlparse(url)
hostname = parsed.hostname
try:
ip = socket.gethostbyname(hostname)
return ip
except:
return "127.0.0.1" # Fallback
async def execute_failover(self, service_name: str) -> bool:
"""Ejecutar procedimiento de failover para un servicio"""
service = self.services[service_name]
if not service.secondary_url:
logger.error(f"No secondary URL configured for {service_name}")
return False
logger.info(f"Initiating failover for {service_name}")
start_time = time.time()
try:
# Ejecutar failover según estrategia
success = False
if service.failover_strategy == FailoverStrategy.ACTIVE_PASSIVE:
success = await self.perform_dns_failover(service_name, service.secondary_url)
elif service.failover_strategy == FailoverStrategy.BLUE_GREEN:
success = await self.perform_blue_green_switch(service_name)
# Registrar evento de failover
recovery_time = time.time() - start_time
event = FailoverEvent(
timestamp=datetime.now(),
service=service_name,
source=service.primary_url,
target=service.secondary_url or "unknown",
reason="Health check failure",
success=success,
recovery_time=recovery_time
)
self.failover_history.append(event)
self.last_failover[service_name] = datetime.now()
# Notificar
await self.send_notification(
f"Failover {'completed' if success else 'failed'} for {service_name}",
event
)
return success
except Exception as e:
logger.error(f"Failover execution failed for {service_name}: {e}")
return False
async def perform_blue_green_switch(self, service_name: str) -> bool:
"""Implementar blue-green deployment switch"""
# Placeholder - implementación específica según infraestructura
logger.info(f"Performing blue-green switch for {service_name}")
return True
async def send_notification(self, message: str, event: FailoverEvent):
"""Enviar notificación de evento de disaster recovery"""
try:
# Slack notification
slack_webhook = self.config.get('notification', {}).get('slack_webhook')
if slack_webhook:
async with aiohttp.ClientSession() as session:
payload = {
"text": f"🚨 Disaster Recovery Alert: {message}",
"attachments": [{
"color": "danger" if not event.success else "good",
"fields": [
{"title": "Service", "value": event.service, "short": True},
{"title": "Recovery Time", "value": f"{event.recovery_time:.2f}s", "short": True},
{"title": "Source", "value": event.source, "short": False},
{"title": "Target", "value": event.target, "short": False}
],
"footer": "Disaster Recovery System",
"ts": event.timestamp.timestamp()
}]
}
await session.post(slack_webhook, json=payload)
except Exception as e:
logger.error(f"Failed to send notification: {e}")
async def monitor_services(self):
"""Loop principal de monitoreo de servicios"""
logger.info("Starting service monitoring loop...")
while True:
try:
for service_name, service in self.services.items():
current_status = await self.check_service_health(service)
# Check database separately if configured
if service_name == 'database':
db_status = await self.check_database_health()
current_status = db_status if db_status != ServiceStatus.UNKNOWN else current_status
previous_status = self.service_states.get(service_name, ServiceStatus.UNKNOWN)
self.service_states[service_name] = current_status
# Handle state transitions
if current_status == ServiceStatus.FAILED:
self.failure_counts[service_name] += 1
if (self.failure_counts[service_name] >=
service.health_checks[0].failure_threshold):
# Execute failover if not already failed over recently
last_failover = self.last_failover.get(service_name)
if (not last_failover or
datetime.now() - last_failover > timedelta(minutes=5)):
await self.execute_failover(service_name)
elif current_status == ServiceStatus.HEALTHY:
if previous_status in [ServiceStatus.FAILED, ServiceStatus.DEGRADED]:
self.recovery_counts[service_name] += 1
# Auto-failback if configured and thresholds met
if (service.auto_failback and
self.recovery_counts[service_name] >=
service.health_checks[0].recovery_threshold):
await self.consider_failback(service_name)
# Reset failure count on recovery
self.failure_counts[service_name] = 0
# Log status changes
if current_status != previous_status:
logger.info(
f"Service {service_name} status changed: "
f"{previous_status.value} -> {current_status.value}"
)
# Wait before next check cycle
await asyncio.sleep(30) # Check every 30 seconds
except Exception as e:
logger.error(f"Error in monitoring loop: {e}")
await asyncio.sleep(10)
async def consider_failback(self, service_name: str):
"""Considerar failback automático después de recovery"""
service = self.services[service_name]
last_failover = self.last_failover.get(service_name)
if (last_failover and
datetime.now() - last_failover > timedelta(seconds=service.failback_delay)):
logger.info(f"Initiating automatic failback for {service_name}")
# Swap primary and secondary URLs for failback
original_primary = service.primary_url
service.primary_url = service.secondary_url
service.secondary_url = original_primary
# Perform DNS failover back to original
success = await self.perform_dns_failover(service_name, service.primary_url)
if success:
logger.info(f"Automatic failback completed for {service_name}")
self.recovery_counts[service_name] = 0
else:
# Revert URLs if failback failed
service.secondary_url = service.primary_url
service.primary_url = original_primary
logger.error(f"Automatic failback failed for {service_name}")
async def get_system_status(self) -> Dict[str, Any]:
"""Obtener estado completo del sistema"""
return {
"timestamp": datetime.now().isoformat(),
"services": {
name: {
"status": status.value,
"failure_count": self.failure_counts.get(name, 0),
"recovery_count": self.recovery_counts.get(name, 0),
"last_failover": (
self.last_failover[name].isoformat()
if name in self.last_failover else None
)
}
for name, status in self.service_states.items()
},
"recent_events": [
{
"timestamp": event.timestamp.isoformat(),
"service": event.service,
"success": event.success,
"recovery_time": event.recovery_time
}
for event in self.failover_history[-10:] # Last 10 events
]
}
async def main():
"""Función principal del sistema de disaster recovery"""
orchestrator = DisasterRecoveryOrchestrator()
await orchestrator.initialize()
# Start monitoring loop
await orchestrator.monitor_services()
if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
logger.info("Disaster Recovery Orchestrator stopped by user")
except Exception as e:
logger.error(f"Fatal error: {e}")
raiseTesting y Validación de Disaster Recovery
El testing regular de procedimientos de disaster recovery es esencial para asegurar que los sistemas funcionarán correctamente durante una emergencia real. Esto incluye chaos engineering, disaster recovery drills, y automated testing de recovery procedures.
#!/bin/bash
## Suite de testing para disaster recovery
set -euo pipefail
## Configuración
TEST_CONFIG="/etc/disaster-recovery/test-config.json"
RESULTS_DIR="/var/log/dr-testing"
NOTIFICATION_WEBHOOK="https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
## Logging
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "${RESULTS_DIR}/dr-test.log"
}
## Test 1: Backup Integrity Testing
test_backup_integrity() {
local test_name="backup_integrity"
local start_time=$(date +%s)
log "Starting backup integrity test..."
# Test database backup restore
local db_backup_file="/backup/postgresql/$(ls /backup/postgresql/ | head -1)"
local test_db_name="test_restore_$(date +%s)"
# Create test database and restore backup
if psql -h localhost -U postgres -c "CREATE DATABASE ${test_db_name};" 2>/dev/null; then
if pg_restore -h localhost -U postgres -d "$test_db_name" "$db_backup_file" 2>/dev/null; then
# Verify data integrity
local row_count=$(psql -h localhost -U postgres -d "$test_db_name" -t -c "SELECT COUNT(*) FROM pg_tables WHERE schemaname = 'public';" 2>/dev/null || echo "0")
if [ "$row_count" -gt 0 ]; then
log "✓ Database backup integrity test passed ($row_count tables restored)"
psql -h localhost -U postgres -c "DROP DATABASE ${test_db_name};" 2>/dev/null
echo "PASS" > "${RESULTS_DIR}/${test_name}.result"
else
log "✗ Database backup integrity test failed (no tables found)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
else
log "✗ Database backup restore failed"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
else
log "✗ Cannot create test database"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Test file system backup
local fs_backup_file="/backup/full/$(ls /backup/full/ | tail -1)"
local test_restore_dir="/tmp/restore_test_$(date +%s)"
mkdir -p "$test_restore_dir"
if gpg --decrypt --batch --passphrase-file /etc/backup/backup.key "$fs_backup_file" | tar -xzf - -C "$test_restore_dir" 2>/dev/null; then
local file_count=$(find "$test_restore_dir" -type f | wc -l)
if [ "$file_count" -gt 100 ]; then # Arbitrary threshold
log "✓ File system backup integrity test passed ($file_count files restored)"
else
log "✗ File system backup integrity test failed (only $file_count files restored)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
rm -rf "$test_restore_dir"
else
log "✗ File system backup decrypt/extract failed"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
local end_time=$(date +%s)
local duration=$((end_time - start_time))
log "Backup integrity test completed in ${duration} seconds"
echo "$duration" > "${RESULTS_DIR}/${test_name}.time"
}
## Test 2: Failover Functionality
test_failover_functionality() {
local test_name="failover_functionality"
local start_time=$(date +%s)
log "Starting failover functionality test..."
# Test database failover (if replica exists)
local primary_db="localhost"
local replica_db="replica.localhost" # Configure as needed
if nc -z "$replica_db" 5432 2>/dev/null; then
log "Testing database failover..."
# Simulate primary failure by blocking connection
iptables -A OUTPUT -d "$primary_db" -p tcp --dport 5432 -j DROP 2>/dev/null || {
log "Warning: Cannot simulate network failure (iptables not available)"
}
# Test connection to replica
sleep 5
if psql -h "$replica_db" -U postgres -c "SELECT 1;" &>/dev/null; then
log "✓ Database failover test passed (replica accessible)"
else
log "✗ Database failover test failed (replica not accessible)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Restore primary connection
iptables -D OUTPUT -d "$primary_db" -p tcp --dport 5432 -j DROP 2>/dev/null || true
else
log "Skipping database failover test (no replica configured)"
fi
# Test application failover via load balancer
local primary_app="http://app.company.com"
local backup_app="http://backup.company.com"
if curl -f "$backup_app/health" &>/dev/null; then
log "✓ Application failover endpoint accessible"
# Simulate primary failure and test failover
# This would typically involve updating DNS or load balancer config
# For testing, we just verify backup is responsive
local response_time=$(curl -o /dev/null -s -w "%{time_total}" "$backup_app/health" || echo "999")
if (( $(echo "$response_time 5.0" | bc -l 2>/dev/null || echo 0) )); then
log "✓ Backup application response time acceptable (${response_time}s)"
else
log "✗ Backup application response time too high (${response_time}s)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
else
log "✗ Backup application not accessible"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
local end_time=$(date +%s)
local duration=$((end_time - start_time))
log "Failover functionality test completed in ${duration} seconds"
echo "$duration" > "${RESULTS_DIR}/${test_name}.time"
}
## Test 3: Recovery Time Objectives (RTO)
test_recovery_time_objectives() {
local test_name="recovery_time_objectives"
local start_time=$(date +%s)
log "Starting RTO test..."
# Define RTO targets (in seconds)
local database_rto=300 # 5 minutes
local application_rto=120 # 2 minutes
local file_system_rto=600 # 10 minutes
# Test database recovery time
local db_recovery_start=$(date +%s)
# Simulate database recovery (restore from backup)
local test_db="rto_test_$(date +%s)"
local latest_backup="/backup/postgresql/$(ls /backup/postgresql/ | tail -1)"
if [ -f "$latest_backup" ]; then
psql -h localhost -U postgres -c "CREATE DATABASE ${test_db};" &>/dev/null
if pg_restore -h localhost -U postgres -d "$test_db" "$latest_backup" &>/dev/null; then
local db_recovery_time=$(($(date +%s) - db_recovery_start))
if [ $db_recovery_time -le $database_rto ]; then
log "✓ Database RTO met (${db_recovery_time}s <= ${database_rto}s)"
else
log "✗ Database RTO exceeded (${db_recovery_time}s > ${database_rto}s)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Cleanup
psql -h localhost -U postgres -c "DROP DATABASE ${test_db};" &>/dev/null
else
log "✗ Database recovery test failed"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
else
log "✗ No database backup found for RTO test"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Test application recovery time (simulate container restart)
if command -v docker &>/dev/null; then
local app_recovery_start=$(date +%s)
# Get a running application container
local container_id=$(docker ps --format "table {{.ID}}\t{{.Names}}" | grep -v CONTAINER | head -1 | cut -f1)
if [ -n "$container_id" ]; then
# Stop and restart container
docker stop "$container_id" &>/dev/null
docker start "$container_id" &>/dev/null
# Wait for health check
local retries=0
while [ $retries -lt 30 ]; do
if docker exec "$container_id" curl -f http://localhost:8080/health &>/dev/null; then
break
fi
sleep 2
((retries++))
done
local app_recovery_time=$(($(date +%s) - app_recovery_start))
if [ $app_recovery_time -le $application_rto ]; then
log "✓ Application RTO met (${app_recovery_time}s <= ${application_rto}s)"
else
log "✗ Application RTO exceeded (${app_recovery_time}s > ${application_rto}s)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
fi
fi
local end_time=$(date +%s)
local duration=$((end_time - start_time))
log "RTO test completed in ${duration} seconds"
echo "$duration" > "${RESULTS_DIR}/${test_name}.time"
}
## Test 4: Recovery Point Objectives (RPO)
test_recovery_point_objectives() {
local test_name="recovery_point_objectives"
local start_time=$(date +%s)
log "Starting RPO test..."
# Define RPO targets (in seconds)
local database_rpo=3600 # 1 hour
local file_system_rpo=86400 # 24 hours
# Check database backup freshness
local latest_db_backup="/backup/postgresql/$(ls -t /backup/postgresql/ | head -1)"
if [ -f "$latest_db_backup" ]; then
local backup_age=$(( $(date +%s) - $(stat -c %Y "$latest_db_backup") ))
if [ $backup_age -le $database_rpo ]; then
log "✓ Database RPO met (backup age: ${backup_age}s <= ${database_rpo}s)"
else
log "✗ Database RPO exceeded (backup age: ${backup_age}s > ${database_rpo}s)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
else
log "✗ No database backup found for RPO test"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Check file system backup freshness
local latest_fs_backup="/backup/full/$(ls -t /backup/full/ | head -1)"
if [ -f "$latest_fs_backup" ]; then
local backup_age=$(( $(date +%s) - $(stat -c %Y "$latest_fs_backup") ))
if [ $backup_age -le $file_system_rpo ]; then
log "✓ File system RPO met (backup age: ${backup_age}s <= ${file_system_rpo}s)"
else
log "✗ File system RPO exceeded (backup age: ${backup_age}s > ${file_system_rpo}s)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
else
log "✗ No file system backup found for RPO test"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Test continuous replication lag (if configured)
if psql -h localhost -U postgres -c "SELECT pg_is_in_recovery();" &>/dev/null; then
local replication_lag=$(psql -h localhost -U postgres -t -c "
SELECT CASE
WHEN pg_is_in_recovery() THEN
EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))::int
ELSE 0
END;
" 2>/dev/null | xargs)
if [ -n "$replication_lag" ] && [ "$replication_lag" -le 60 ]; then # 1 minute lag threshold
log "✓ Database replication lag acceptable (${replication_lag}s)"
else
log "✗ Database replication lag too high (${replication_lag}s)"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
fi
local end_time=$(date +%s)
local duration=$((end_time - start_time))
log "RPO test completed in ${duration} seconds"
echo "$duration" > "${RESULTS_DIR}/${test_name}.time"
}
## Test 5: Chaos Engineering - Network Failures
test_network_chaos() {
local test_name="network_chaos"
local start_time=$(date +%s)
log "Starting network chaos test..."
# Simulate various network failures
local test_scenarios=(
"latency:100ms"
"packet_loss:1%"
"connection_timeout:5s"
)
for scenario in "${test_scenarios[@]}"; do
local chaos_type=$(echo "$scenario" | cut -d: -f1)
local chaos_value=$(echo "$scenario" | cut -d: -f2)
log "Testing chaos scenario: $chaos_type = $chaos_value"
case "$chaos_type" in
"latency")
# Add network latency
tc qdisc add dev eth0 root netem delay "$chaos_value" 2>/dev/null || {
log "Warning: Cannot add network latency (tc not available)"
continue
}
;;
"packet_loss")
# Add packet loss
tc qdisc add dev eth0 root netem loss "$chaos_value" 2>/dev/null || {
log "Warning: Cannot add packet loss (tc not available)"
continue
}
;;
"connection_timeout")
# Simulate connection timeout via iptables
iptables -A OUTPUT -p tcp --dport 80 -j DROP 2>/dev/null || {
log "Warning: Cannot simulate connection timeout (iptables not available)"
continue
}
;;
esac
# Test application resilience during chaos
sleep 10
local health_check_passed=false
if curl -f --max-time 10 http://localhost:8080/health &>/dev/null; then
health_check_passed=true
fi
# Cleanup chaos
case "$chaos_type" in
"latency"|"packet_loss")
tc qdisc del dev eth0 root 2>/dev/null || true
;;
"connection_timeout")
iptables -D OUTPUT -p tcp --dport 80 -j DROP 2>/dev/null || true
;;
esac
if [ "$health_check_passed" = true ]; then
log "✓ Application survived chaos scenario: $scenario"
else
log "✗ Application failed during chaos scenario: $scenario"
echo "FAIL" > "${RESULTS_DIR}/${test_name}.result"
fi
# Wait for recovery
sleep 5
done
local end_time=$(date +%s)
local duration=$((end_time - start_time))
log "Network chaos test completed in ${duration} seconds"
echo "$duration" > "${RESULTS_DIR}/${test_name}.time"
}
## Generate test report
generate_test_report() {
local report_file="${RESULTS_DIR}/dr-test-report-$(date +%Y%m%d_%H%M%S).json"
local total_tests=0
local passed_tests=0
log "Generating test report..."
cat > "$report_file" <EOF
{
"test_run": {
"timestamp": "$(date -Iseconds)",
"duration": "$(($(date +%s) - START_TIME)) seconds"
},
"tests": {
EOF
local first_test=true
for test_result in "${RESULTS_DIR}"/*.result; do
if [ -f "$test_result" ]; then
local test_name=$(basename "$test_result" .result)
local result=$(cat "$test_result")
local time_file="${RESULTS_DIR}/${test_name}.time"
local test_time="unknown"
if [ -f "$time_file" ]; then
test_time=$(cat "$time_file")
fi
if [ "$first_test" = false ]; then
echo "," >> "$report_file"
fi
first_test=false
cat >> "$report_file" <EOF
"${test_name}": {
"result": "${result}",
"duration": "${test_time} seconds"
}
EOF
((total_tests++))
if [ "$result" = "PASS" ] || [ ! -f "$test_result" ]; then
((passed_tests++))
fi
fi
done
cat >> "$report_file" <EOF
},
"summary": {
"total_tests": ${total_tests},
"passed_tests": ${passed_tests},
"failed_tests": $((total_tests - passed_tests)),
"success_rate": $(( passed_tests * 100 / total_tests ))%"
}
}
EOF
log "Test report generated: $report_file"
# Send notification
if [ -n "$NOTIFICATION_WEBHOOK" ]; then
local color="good"
if [ $passed_tests -lt $total_tests ]; then
color="danger"
fi
curl -X POST "$NOTIFICATION_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{
\"text\": \"🧪 Disaster Recovery Test Completed\",
\"attachments\": [{
\"color\": \"${color}\",
\"fields\": [
{\"title\": \"Total Tests\", \"value\": \"${total_tests}\", \"short\": true},
{\"title\": \"Passed\", \"value\": \"${passed_tests}\", \"short\": true},
{\"title\": \"Success Rate\", \"value\": \"$(( passed_tests * 100 / total_tests ))%\", \"short\": true}
]
}]
}" &>/dev/null || log "Failed to send notification"
fi
}
## Main execution
main() {
local START_TIME=$(date +%s)
mkdir -p "$RESULTS_DIR"
log "Starting disaster recovery testing suite..."
# Clean previous results
rm -f "${RESULTS_DIR}"/*.result "${RESULTS_DIR}"/*.time
# Run all tests
test_backup_integrity
test_failover_functionality
test_recovery_time_objectives
test_recovery_point_objectives
test_network_chaos
# Generate report
generate_test_report
log "Disaster recovery testing completed"
}
## Execute main function
main "$@"Mejores Prácticas y Compliance
Estrategias de Retention y Legal Compliance
El manejo de retention policies y compliance legal requiere consideración cuidadosa de regulaciones como GDPR, HIPAA, SOX, y requisitos específicos de industria. Esto incluye data lifecycle management, secure deletion, y audit trails.
Encryption y Security en Backups
La seguridad de backups incluye encryption at rest y in transit, key management, access controls, y secure storage locations. Los backups son a menudo targets atractivos para attackers debido a que contienen datos históricos completos.
Cost Optimization para Backup Storage
La optimización de costos para almacenamiento de backup incluye tiering strategies, compression algorithms, deduplication technologies, y lifecycle policies que mueven datos entre different storage classes basado en age y access patterns.
Conclusión
El backup y recuperación a nivel de sistema representa mucho más que una simple copia de seguridad de datos; constituye la base fundamental sobre la cual se construye la resiliencia empresarial moderna. En un ecosistema tecnológico donde los datos son el activo más valioso y donde minutes of downtime pueden traducirse en millions of dollars in losses, las estrategias comprehensivas de backup y disaster recovery se han convertido en imperativos críticos de negocio.
La evolución hacia architectures cloud-native, microservices, y distributed systems ha transformado radicalmente los requisitos y capabilities de backup y recovery. Ya no es suficiente realizar backup nightly y esperar recovery times de horas; las organizaciones modernas requieren RPOs medidos en seconds y RTOs medidos en minutes, con capabilities de automatic failover y seamless disaster recovery.
La implementación exitosa requiere adoptar un approach holístico que combine tecnologías avanzadas, procesos bien definidos, testing rigorous, y cultural transformation. Las organizaciones que invierten en desarrollar mature backup y disaster recovery capabilities no solo protegen sus assets críticos sino que también obtienen ventajas competitivas significativas en términos de business continuity, customer confidence, y regulatory compliance.
El futuro del backup y disaster recovery continuará evolucionando con advances in AI-driven predictive failure detection, quantum-resistant encryption, edge computing backup strategies, y fully automated disaster recovery orchestration. La inversión en dominar estas technologies y practices proporcionará dividendos substanciales en términos de business resilience, operational efficiency, y strategic advantage.
La clave del éxito radica en reconocer que backup y disaster recovery no son projects one-time sino ongoing processes que requieren continuous improvement, regular testing, y adaptation a changing business requirements y threat landscapes. Con proper implementation y maintenance, estos systems proporcionan la confidence y security necesaria para permitir innovation y growth sin compromising business continuity.