312 lines
8.0 KiB
Markdown
312 lines
8.0 KiB
Markdown
# Data Retention Service
|
|
|
|
A lightweight, standalone microservice responsible for automated data cleanup based on tenant retention policies.
|
|
|
|
## Overview
|
|
|
|
This service runs as a separate Docker container and performs the following functions:
|
|
|
|
- **Automated Cleanup**: Daily scheduled cleanup at 2:00 AM UTC
|
|
- **Tenant-Aware**: Respects individual tenant retention policies
|
|
- **Lightweight**: Minimal resource footprint (~64-128MB RAM)
|
|
- **Resilient**: Continues operation even if individual tenant cleanups fail
|
|
- **Logged**: Comprehensive logging and health monitoring
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ Main Backend │ │ Data Retention │ │ PostgreSQL │
|
|
│ Container │ │ Service │ │ Database │
|
|
│ │ │ │ │ │
|
|
│ • API Endpoints │ │ • Cron Jobs │◄──►│ • tenant data │
|
|
│ • Business Logic│ │ • Data Cleanup │ │ • detections │
|
|
│ • Rate Limiting │ │ • Health Check │ │ • heartbeats │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
## Features
|
|
|
|
### 🕒 Scheduled Operations
|
|
- Runs daily at 2:00 AM UTC via cron job
|
|
- Configurable immediate cleanup for development/testing
|
|
- Graceful shutdown handling
|
|
|
|
### 🏢 Multi-Tenant Support
|
|
- Processes all active tenants
|
|
- Respects individual retention policies:
|
|
- `-1` = Unlimited retention (no cleanup)
|
|
- `N` = Delete data older than N days
|
|
- Default: 90 days if not specified
|
|
|
|
### 🧹 Data Cleanup
|
|
- **Drone Detections**: Historical detection records
|
|
- **Heartbeats**: Device connectivity logs
|
|
- **Security Logs**: Audit trail entries (if applicable)
|
|
|
|
### 📊 Monitoring & Health
|
|
- Built-in health checks for Docker
|
|
- Memory usage monitoring
|
|
- Cleanup statistics tracking
|
|
- Error logging with tenant context
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
```bash
|
|
# Database Connection
|
|
DB_HOST=postgres # Database host
|
|
DB_PORT=5432 # Database port
|
|
DB_NAME=drone_detection # Database name
|
|
DB_USER=postgres # Database user
|
|
DB_PASSWORD=password # Database password
|
|
|
|
# Service Settings
|
|
NODE_ENV=production # Environment mode
|
|
IMMEDIATE_CLEANUP=false # Run cleanup on startup
|
|
LOG_LEVEL=info # Logging level
|
|
```
|
|
|
|
### Docker Compose Integration
|
|
|
|
```yaml
|
|
data-retention:
|
|
build:
|
|
context: ./data-retention-service
|
|
container_name: drone-detection-data-retention
|
|
restart: unless-stopped
|
|
environment:
|
|
DB_HOST: postgres
|
|
DB_PORT: 5432
|
|
DB_NAME: drone_detection
|
|
DB_USER: postgres
|
|
DB_PASSWORD: your_secure_password
|
|
depends_on:
|
|
postgres:
|
|
condition: service_healthy
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 128M
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Start with Docker Compose
|
|
|
|
```bash
|
|
# Start all services including data retention
|
|
docker-compose up -d
|
|
|
|
# Start only data retention service
|
|
docker-compose up -d data-retention
|
|
|
|
# View logs
|
|
docker-compose logs -f data-retention
|
|
```
|
|
|
|
### Manual Container Build
|
|
|
|
```bash
|
|
# Build the container
|
|
cd data-retention-service
|
|
docker build -t data-retention-service .
|
|
|
|
# Run the container
|
|
docker run -d \
|
|
--name data-retention \
|
|
--env-file .env \
|
|
--network drone-network \
|
|
data-retention-service
|
|
```
|
|
|
|
### Development Mode
|
|
|
|
```bash
|
|
# Install dependencies
|
|
npm install
|
|
|
|
# Run with immediate cleanup
|
|
IMMEDIATE_CLEANUP=true npm start
|
|
|
|
# Run in development mode
|
|
npm run dev
|
|
```
|
|
|
|
## Logging Output
|
|
|
|
### Startup
|
|
```
|
|
🗂️ Starting Data Retention Service...
|
|
📅 Environment: production
|
|
💾 Database: postgres:5432/drone_detection
|
|
✅ Database connection established
|
|
⏰ Scheduled cleanup: Daily at 2:00 AM UTC
|
|
✅ Data Retention Service started successfully
|
|
```
|
|
|
|
### Cleanup Operation
|
|
```
|
|
🧹 Starting data retention cleanup...
|
|
⏰ Cleanup started at: 2024-09-23T02:00:00.123Z
|
|
🏢 Found 5 active tenants to process
|
|
🧹 Cleaning tenant acme-corp - removing data older than 90 days
|
|
✅ Tenant acme-corp: Deleted 1250 detections, 4500 heartbeats, 89 logs
|
|
⏭️ Skipping tenant enterprise-unlimited - unlimited retention
|
|
✅ Data retention cleanup completed
|
|
⏱️ Duration: 2347ms
|
|
📊 Deleted: 2100 detections, 8900 heartbeats, 156 logs
|
|
```
|
|
|
|
### Health Monitoring
|
|
```
|
|
💚 Health Check - Uptime: 3600s, Memory: 45MB, Last Cleanup: 2024-09-23T02:00:00.123Z
|
|
```
|
|
|
|
## API Integration
|
|
|
|
The main backend provides endpoints to interact with retention policies:
|
|
|
|
```bash
|
|
# Get current tenant limits and retention info
|
|
GET /api/tenant/limits
|
|
|
|
# Preview what would be deleted
|
|
GET /api/tenant/data-retention/preview
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
### Tenant-Level Errors
|
|
- Service continues if individual tenant cleanup fails
|
|
- Errors logged with tenant context
|
|
- Failed tenants skipped, others processed normally
|
|
|
|
### Service-Level Errors
|
|
- Database connection issues cause service restart
|
|
- Health checks detect and report issues
|
|
- Graceful shutdown on container stop signals
|
|
|
|
### Example Error Log
|
|
```
|
|
❌ Error cleaning tenant problematic-tenant: SequelizeTimeoutError: Query timeout
|
|
⚠️ Errors encountered: 1
|
|
- problematic-tenant: Query timeout
|
|
```
|
|
|
|
## Performance
|
|
|
|
### Resource Usage
|
|
- **Memory**: 64-128MB typical usage
|
|
- **CPU**: Minimal, only during cleanup operations
|
|
- **Storage**: Logs rotate automatically
|
|
- **Network**: Database queries only
|
|
|
|
### Cleanup Performance
|
|
- Batch operations for efficiency
|
|
- Indexed database queries on timestamp fields
|
|
- Parallel tenant processing where possible
|
|
- Configurable batch sizes for large datasets
|
|
|
|
## Security
|
|
|
|
### Database Access
|
|
- Read/write access only to required tables
|
|
- Connection pooling with limits
|
|
- Prepared statements prevent SQL injection
|
|
|
|
### Container Security
|
|
- Non-root user execution
|
|
- Minimal base image (node:18-alpine)
|
|
- No exposed ports
|
|
- Isolated network access
|
|
|
|
## Monitoring
|
|
|
|
### Health Checks
|
|
```bash
|
|
# Docker health check
|
|
docker exec data-retention node healthcheck.js
|
|
|
|
# Container status
|
|
docker-compose ps data-retention
|
|
|
|
# Service logs
|
|
docker-compose logs -f data-retention
|
|
```
|
|
|
|
### Metrics
|
|
- Cleanup duration and frequency
|
|
- Records deleted per tenant
|
|
- Memory usage over time
|
|
- Error rates and types
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
**Service won't start**
|
|
```bash
|
|
# Check database connectivity
|
|
docker-compose logs postgres
|
|
docker-compose logs data-retention
|
|
|
|
# Verify environment variables
|
|
docker-compose config
|
|
```
|
|
|
|
**Cleanup not running**
|
|
```bash
|
|
# Check cron schedule
|
|
docker exec data-retention ps aux | grep cron
|
|
|
|
# Force immediate cleanup
|
|
docker exec data-retention node -e "
|
|
const service = require('./index.js');
|
|
service.performCleanup();
|
|
"
|
|
```
|
|
|
|
**High memory usage**
|
|
```bash
|
|
# Check cleanup frequency
|
|
docker stats data-retention
|
|
|
|
# Review tenant data volumes
|
|
docker exec data-retention node -e "
|
|
const { getModels } = require('./database');
|
|
// Check tenant data sizes
|
|
"
|
|
```
|
|
|
|
### Configuration Validation
|
|
|
|
```bash
|
|
# Test database connection
|
|
docker exec data-retention node healthcheck.js
|
|
|
|
# Verify tenant policies
|
|
docker exec -it data-retention node -e "
|
|
const { getModels } = require('./database');
|
|
(async () => {
|
|
const { Tenant } = await getModels();
|
|
const tenants = await Tenant.findAll();
|
|
console.log(tenants.map(t => ({
|
|
slug: t.slug,
|
|
retention: t.features?.data_retention_days
|
|
})));
|
|
})();
|
|
"
|
|
```
|
|
|
|
## Migration from Integrated Service
|
|
|
|
If upgrading from a version where data retention was part of the main backend:
|
|
|
|
1. **Deploy new container**: Add data retention service to docker-compose.yml
|
|
2. **Verify operation**: Check logs for successful startup and database connection
|
|
3. **Remove old code**: The integrated service code is automatically disabled
|
|
4. **Monitor transition**: Ensure cleanup operations continue normally
|
|
|
|
The service is designed to be backward compatible and will work with existing tenant configurations without changes. |