Fix jwt-token
This commit is contained in:
312
data-retention-service/README.md
Normal file
312
data-retention-service/README.md
Normal file
@@ -0,0 +1,312 @@
|
||||
# Data Retention Service
|
||||
|
||||
A lightweight, standalone microservice responsible for automated data cleanup based on tenant retention policies.
|
||||
|
||||
## Overview
|
||||
|
||||
This service runs as a separate Docker container and performs the following functions:
|
||||
|
||||
- **Automated Cleanup**: Daily scheduled cleanup at 2:00 AM UTC
|
||||
- **Tenant-Aware**: Respects individual tenant retention policies
|
||||
- **Lightweight**: Minimal resource footprint (~64-128MB RAM)
|
||||
- **Resilient**: Continues operation even if individual tenant cleanups fail
|
||||
- **Logged**: Comprehensive logging and health monitoring
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Main Backend │ │ Data Retention │ │ PostgreSQL │
|
||||
│ Container │ │ Service │ │ Database │
|
||||
│ │ │ │ │ │
|
||||
│ • API Endpoints │ │ • Cron Jobs │◄──►│ • tenant data │
|
||||
│ • Business Logic│ │ • Data Cleanup │ │ • detections │
|
||||
│ • Rate Limiting │ │ • Health Check │ │ • heartbeats │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
### 🕒 Scheduled Operations
|
||||
- Runs daily at 2:00 AM UTC via cron job
|
||||
- Configurable immediate cleanup for development/testing
|
||||
- Graceful shutdown handling
|
||||
|
||||
### 🏢 Multi-Tenant Support
|
||||
- Processes all active tenants
|
||||
- Respects individual retention policies:
|
||||
- `-1` = Unlimited retention (no cleanup)
|
||||
- `N` = Delete data older than N days
|
||||
- Default: 90 days if not specified
|
||||
|
||||
### 🧹 Data Cleanup
|
||||
- **Drone Detections**: Historical detection records
|
||||
- **Heartbeats**: Device connectivity logs
|
||||
- **Security Logs**: Audit trail entries (if applicable)
|
||||
|
||||
### 📊 Monitoring & Health
|
||||
- Built-in health checks for Docker
|
||||
- Memory usage monitoring
|
||||
- Cleanup statistics tracking
|
||||
- Error logging with tenant context
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Database Connection
|
||||
DB_HOST=postgres # Database host
|
||||
DB_PORT=5432 # Database port
|
||||
DB_NAME=drone_detection # Database name
|
||||
DB_USER=postgres # Database user
|
||||
DB_PASSWORD=password # Database password
|
||||
|
||||
# Service Settings
|
||||
NODE_ENV=production # Environment mode
|
||||
IMMEDIATE_CLEANUP=false # Run cleanup on startup
|
||||
LOG_LEVEL=info # Logging level
|
||||
```
|
||||
|
||||
### Docker Compose Integration
|
||||
|
||||
```yaml
|
||||
data-retention:
|
||||
build:
|
||||
context: ./data-retention-service
|
||||
container_name: drone-detection-data-retention
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
DB_HOST: postgres
|
||||
DB_PORT: 5432
|
||||
DB_NAME: drone_detection
|
||||
DB_USER: postgres
|
||||
DB_PASSWORD: your_secure_password
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 128M
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Start with Docker Compose
|
||||
|
||||
```bash
|
||||
# Start all services including data retention
|
||||
docker-compose up -d
|
||||
|
||||
# Start only data retention service
|
||||
docker-compose up -d data-retention
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f data-retention
|
||||
```
|
||||
|
||||
### Manual Container Build
|
||||
|
||||
```bash
|
||||
# Build the container
|
||||
cd data-retention-service
|
||||
docker build -t data-retention-service .
|
||||
|
||||
# Run the container
|
||||
docker run -d \
|
||||
--name data-retention \
|
||||
--env-file .env \
|
||||
--network drone-network \
|
||||
data-retention-service
|
||||
```
|
||||
|
||||
### Development Mode
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install
|
||||
|
||||
# Run with immediate cleanup
|
||||
IMMEDIATE_CLEANUP=true npm start
|
||||
|
||||
# Run in development mode
|
||||
npm run dev
|
||||
```
|
||||
|
||||
## Logging Output
|
||||
|
||||
### Startup
|
||||
```
|
||||
🗂️ Starting Data Retention Service...
|
||||
📅 Environment: production
|
||||
💾 Database: postgres:5432/drone_detection
|
||||
✅ Database connection established
|
||||
⏰ Scheduled cleanup: Daily at 2:00 AM UTC
|
||||
✅ Data Retention Service started successfully
|
||||
```
|
||||
|
||||
### Cleanup Operation
|
||||
```
|
||||
🧹 Starting data retention cleanup...
|
||||
⏰ Cleanup started at: 2024-09-23T02:00:00.123Z
|
||||
🏢 Found 5 active tenants to process
|
||||
🧹 Cleaning tenant acme-corp - removing data older than 90 days
|
||||
✅ Tenant acme-corp: Deleted 1250 detections, 4500 heartbeats, 89 logs
|
||||
⏭️ Skipping tenant enterprise-unlimited - unlimited retention
|
||||
✅ Data retention cleanup completed
|
||||
⏱️ Duration: 2347ms
|
||||
📊 Deleted: 2100 detections, 8900 heartbeats, 156 logs
|
||||
```
|
||||
|
||||
### Health Monitoring
|
||||
```
|
||||
💚 Health Check - Uptime: 3600s, Memory: 45MB, Last Cleanup: 2024-09-23T02:00:00.123Z
|
||||
```
|
||||
|
||||
## API Integration
|
||||
|
||||
The main backend provides endpoints to interact with retention policies:
|
||||
|
||||
```bash
|
||||
# Get current tenant limits and retention info
|
||||
GET /api/tenant/limits
|
||||
|
||||
# Preview what would be deleted
|
||||
GET /api/tenant/data-retention/preview
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Tenant-Level Errors
|
||||
- Service continues if individual tenant cleanup fails
|
||||
- Errors logged with tenant context
|
||||
- Failed tenants skipped, others processed normally
|
||||
|
||||
### Service-Level Errors
|
||||
- Database connection issues cause service restart
|
||||
- Health checks detect and report issues
|
||||
- Graceful shutdown on container stop signals
|
||||
|
||||
### Example Error Log
|
||||
```
|
||||
❌ Error cleaning tenant problematic-tenant: SequelizeTimeoutError: Query timeout
|
||||
⚠️ Errors encountered: 1
|
||||
- problematic-tenant: Query timeout
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
### Resource Usage
|
||||
- **Memory**: 64-128MB typical usage
|
||||
- **CPU**: Minimal, only during cleanup operations
|
||||
- **Storage**: Logs rotate automatically
|
||||
- **Network**: Database queries only
|
||||
|
||||
### Cleanup Performance
|
||||
- Batch operations for efficiency
|
||||
- Indexed database queries on timestamp fields
|
||||
- Parallel tenant processing where possible
|
||||
- Configurable batch sizes for large datasets
|
||||
|
||||
## Security
|
||||
|
||||
### Database Access
|
||||
- Read/write access only to required tables
|
||||
- Connection pooling with limits
|
||||
- Prepared statements prevent SQL injection
|
||||
|
||||
### Container Security
|
||||
- Non-root user execution
|
||||
- Minimal base image (node:18-alpine)
|
||||
- No exposed ports
|
||||
- Isolated network access
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
```bash
|
||||
# Docker health check
|
||||
docker exec data-retention node healthcheck.js
|
||||
|
||||
# Container status
|
||||
docker-compose ps data-retention
|
||||
|
||||
# Service logs
|
||||
docker-compose logs -f data-retention
|
||||
```
|
||||
|
||||
### Metrics
|
||||
- Cleanup duration and frequency
|
||||
- Records deleted per tenant
|
||||
- Memory usage over time
|
||||
- Error rates and types
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Service won't start**
|
||||
```bash
|
||||
# Check database connectivity
|
||||
docker-compose logs postgres
|
||||
docker-compose logs data-retention
|
||||
|
||||
# Verify environment variables
|
||||
docker-compose config
|
||||
```
|
||||
|
||||
**Cleanup not running**
|
||||
```bash
|
||||
# Check cron schedule
|
||||
docker exec data-retention ps aux | grep cron
|
||||
|
||||
# Force immediate cleanup
|
||||
docker exec data-retention node -e "
|
||||
const service = require('./index.js');
|
||||
service.performCleanup();
|
||||
"
|
||||
```
|
||||
|
||||
**High memory usage**
|
||||
```bash
|
||||
# Check cleanup frequency
|
||||
docker stats data-retention
|
||||
|
||||
# Review tenant data volumes
|
||||
docker exec data-retention node -e "
|
||||
const { getModels } = require('./database');
|
||||
// Check tenant data sizes
|
||||
"
|
||||
```
|
||||
|
||||
### Configuration Validation
|
||||
|
||||
```bash
|
||||
# Test database connection
|
||||
docker exec data-retention node healthcheck.js
|
||||
|
||||
# Verify tenant policies
|
||||
docker exec -it data-retention node -e "
|
||||
const { getModels } = require('./database');
|
||||
(async () => {
|
||||
const { Tenant } = await getModels();
|
||||
const tenants = await Tenant.findAll();
|
||||
console.log(tenants.map(t => ({
|
||||
slug: t.slug,
|
||||
retention: t.features?.data_retention_days
|
||||
})));
|
||||
})();
|
||||
"
|
||||
```
|
||||
|
||||
## Migration from Integrated Service
|
||||
|
||||
If upgrading from a version where data retention was part of the main backend:
|
||||
|
||||
1. **Deploy new container**: Add data retention service to docker-compose.yml
|
||||
2. **Verify operation**: Check logs for successful startup and database connection
|
||||
3. **Remove old code**: The integrated service code is automatically disabled
|
||||
4. **Monitor transition**: Ensure cleanup operations continue normally
|
||||
|
||||
The service is designed to be backward compatible and will work with existing tenant configurations without changes.
|
||||
Reference in New Issue
Block a user