Spendplane Global
Provided by Spendplane Docs Assistant

Fallback strategies for mission-critical LLM downtime

How do I route v0 traffic to Llama-3?
# Fallback Strategies for Mission-Critical LLM Downtime ## Executive Summary This technical deep-dive provides a comprehensive framework for designing resilient fallback strategies when mission-critical Large Language Models (LLMs) experience downtime, latency spikes, or service degradation. The document covers architectural patterns, implementation details, and operational best practices for maintaining system availability during provider outages. --- ## Table of Contents 1. [Problem Statement & Impact Analysis](#problem-statement--impact-analysis) 2. [Fallback Strategy Taxonomy](#fallback-strategy-taxonomy) 3. [Level 1: Pre-Request Fallbacks (Preventive)](#level-1-pre-request-fallbacks-preventive) 4. [Level 2: Runtime Fallbacks (Reactive)](#level-2-runtime-fallbacks-reactive) 5. [Level 3: Degraded Mode Strategies](#level-3-degraded-mode-strategies) 6. [Level 4: Advanced Recovery Mechanisms](#level-4-advanced-recovery-mechanisms) 7. [Integration Framework](#integration-framework-complete-fallback-system) 8. [Performance Benchmarks & Metrics](#performance-benchmarks-and-metrics) --- ## Problem Statement & Impact Analysis ### Critical Failure Scenarios | Scenario | Description | Business Impact | Frequency Estimate | |----------|-------------|-----------------|-------------------| | **Total Model Outage** | Provider API completely unavailable (HTTP 5xx, DNS failure) | Complete feature paralysis | ~4 hours/month per provider | | **Rate Limit Exhaustion** | Token/request limits exceeded (HTTP 429) | Partial service degradation | Variable by traffic patterns | | **Latency Spikes** | Response times exceed SLA (>10s) | Poor UX, timeouts | ~15% of requests during peak | | **Context Window Errors** | Token limit exceeded mid-conversation | Truncated responses, context loss | High for long-running sessions | | **Partial Failures** | Some endpoints/models unavailable | Inconsistent behavior | ~2-3 hours/month | ### Quantified Impact Assessment ```python # Example impact calculation model impact_matrix = { 'downtime_cost_per_hour': 10_000, # Revenue + reputation cost 'user_churn_rate_during_outage': 0.05, # 5% session abandonment per hour 'support_ticket_surge': 3.2x, # 3.2x normal ticket volume 'recovery_time_objective': '15 minutes' | Target RTO } ``` --- ## Fallback Strategy Taxonomy ### Strategic Framework: Defense in Depth ``` ┌─────────────────────────────────────────────────────────────────┐ │ FALLBACK STRATEGY LAYERS │ ├─────────────────────────────────────────────────────────────────┤ │ Layer 1: PREVENTIVE (Before Request) │ │ ├── Intelligent Caching │ │ ├── Request Validation & Normalization │ │ └── Predictive Pre-warming │ ├─────────────────────────────────────────────────────────────────┤ │ Layer 2: REACTIVE (During Request) │ │ ├── Circuit Breaker Pattern │ │ ├── Multi-Provider Fallback Hierarchy │ │ └── Retry with Exponential Backoff │ ├─────────────────────────────────────────────────────────────────┤ │ Layer 3: DEGRADATION (Service Continuity) │ │ ├── Graceful Degradation Tiers │ │ ├── Template-Based Responses │ │ └── Asynchronous Processing Queue │ ├─────────────────────────────────────────────────────────────────┤ │ Layer 4: RECOVERY (Post-Outage) │ │ ├── Health Monitoring & Auto-Recovery │ │ ├── Model Warm-up Procedures │ │ └── Human-in-the-Loop Handoff │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## Level 1: Pre-Request Fallbacks (Preventive) ### Strategy A: Intelligent Caching Layer **Purpose**: Reduce dependency on live LLM calls by caching deterministic responses. #### Implementation Architecture ```python from hashlib import sha256 import redis import json class LLMCacheLayer: def __init__(self, cache_ttl=3600, similarity_threshold=0.8): self.cache = redis.Redis() self.ttl = cache_ttl def generate_cache_key(self, prompt: str) -> str: """Generate deterministic cache key""" normalized = prompt.lower().strip() return sha256(normalized.encode()).hexdigest() def get_cached_response(self, prompt: str) -> dict | None: """Retrieve cached response with metadata""" key = self.generate_cache_key(prompt) cached = self.cache.get(key) if cached: data = json.loads(cached) return { 'response': data['response'], 'source': 'CACHE', 'confidence': data['similarity_score'] } return None def cache_response(self, prompt: str, response: dict): """Store response with metadata""" key = self.generate_cache_key(prompt) self.cache.setex( key, self.ttl, json.dumps({ 'response': response, 'timestamp': datetime.now().isoformat(), 'similarity_score': 1.0 }) ) # Usage example cache = LLMCacheLayer() cached_result = cache.get_cached_response(user_prompt) ``` #### Cache Strategy Types | Type | Description | Use Case | Hit Rate Target | |------|-------------|----------|-----------------| | **Exact Match** | SHA256 hash of prompt | FAQ, common queries | 40-60% | | **Semantic Similarity** | Embedding-based lookup | Conversational context | 30-50% | | **Session Cache** | Per-user conversation history | Multi-turn dialogues | 70-85% | --- ## Level 2: Runtime Fallbacks (Reactive) ### Strategy B: Circuit Breaker Pattern Implementation **Purpose**: Prevent cascade failures by temporarily blocking requests to unhealthy providers. #### State Machine Design ``` ┌─────────────┐ │ CLOSED │ ← Normal operation, requests flow through └──────┬──────┘ │ Failure threshold exceeded ▼ ┌─────────────┐ │ OPEN │ ← Fail fast, no requests to provider └──────┬──────┘ │ Timeout period elapsed ▼ ┌─────────────┐ │ HALF-OPEN │ ← Test recovery with limited traffic └──────┬──────┘ │ Success threshold met │ Failure occurs ◄│ │► │ │ └────┘ ``` #### Implementation Code ```python from enum import Enum import time from functools import wraps class CircuitState(Enum): CLOSED = "closed" # Normal operation OPEN = "open" # Fail fast, no requests HALF_OPEN = "half_open" # Testing recovery class LLMSafetyCircuit: def __init__(self, failure_threshold=5, success_threshold=3, timeout_seconds=60): self.state = CircuitState.CLOSED self.failure_count = 0 self.success_count = 0 self.last_failure_time = None self.thresholds = { 'failure': failure_threshold, 'success': success_threshold, 'timeout': timeout_seconds } def call(self, llm_request_func): """Decorator for protected LLM calls""" @wraps(llm_request_func) async def wrapper(*args, **kwargs): if self.state == CircuitState.OPEN: # Check if we should test recovery if time.time() - self.last_failure_time > self.thresholds['timeout']: self.state = CircuitState.HALF_OPEN print("Circuit HALF-OPEN: Testing recovery") else: return await self._execute_fallback(*args, **kwargs) try: result = await llm_request_func(*args, **kwargs) self._on_success() return result except Exception as e: self._on_failure(e) raise FallbackTriggered(f"Circuit triggered fallback", original_error=e) return wrapper def _on_success(self): if self.state == CircuitState.HALF_OPEN: self.success_count += 1 if self.success_count >= self.thresholds['success']: self._reset_circuit() else: self.failure_count = max(0, self.failure_count - 1) def _on_failure(self, error): self.failure_count += 1 self.last_failure_time = time.time() if self.state == CircuitState.HALF_OPEN: self._open_circuit(error) elif self.failure_count >= self.thresholds['failure']: self._open_circuit(error) def _open_circuit(self, error): self.state = CircuitState.OPEN print(f"⚠️ CIRCUIT OPENED: {error}") ``` ### Strategy C: Multi-Provider Fallback Hierarchy **Purpose**: Automatically route requests to alternative providers when primary fails. #### Provider Priority Matrix | Tier | Provider Type | Cost/1K tokens | Latency P95 | Use Case | |------|---------------|----------------|-------------|----------| | **Primary (0)** | GPT-4 Turbo / Claude Opus | $0.01 - $0.02 | ~800ms | Default routing | | **Fallback 1 (1)** | GPT-3.5 / Claude Sonnet | $0.001 - $0.006 | ~400ms | High-volume, lower complexity | | **Fallback 2 (2)** | Open-source (Llama, Mistral) via API | $0.0005 - $0.002 | ~600ms | Cost-sensitive fallback | | **Fallback 3 (3)** | Local inference (Ollama, vLLM) | $0.00 + infra | ~1-2s | Offline/air-gapped scenarios | #### Implementation Code ```python from typing import List, Optional, Callable class ProviderConfig: def __init__(self, provider_name: str, api_call: Callable, priority: int = 0, cost_per_token: float = 0.0, max_retries: int = 3): self.provider_name = provider_name self.api_call = api_call self.priority = priority self.cost_per_token = cost_per_token self.max_retries = max_retries class MultiProviderRouter: def __init__(self, providers: List[ProviderConfig]): # Sort by priority (lower number = higher priority) self.providers = sorted(providers, key=lambda p: p.priority) async def call_with_fallback(self, prompt: str, config: dict): """Execute request with automatic provider fallback""" last_error = None for provider in self.providers: try: print(f"🔄 Attempting: {provider.provider_name}") response = await provider.api_call( prompt=prompt, config=config, max_retries=provider.max_retries ) return { 'response': response, 'provider': provider.provider_name, 'cost_estimate': self._calculate_cost(response, provider), 'fallback_count': 0 if provider == self.providers[0] else self.providers.index(provider) + 1 } except Exception as e: last_error = e print(f"❌ {provider.provider_name} failed: {str(e)}") # All providers exhausted raise ExhaustedProvidersError( f"All {len(self.providers)} providers failed", errors=[last_error] ) # Example provider configuration providers = [ ProviderConfig("GPT-4-Turbo", call_gpt4, priority=0), ProviderConfig("Claude-3-Sonnet", call_claude, priority=1), ProviderConfig("Gemini-Pro", call_gemini, priority=2), ProviderConfig("Local-Ollama", call_local_llm, priority=3), # Offline fallback ] router = MultiProviderRouter(providers) ``` --- ## Level 3: Degraded Mode Strategies (Graceful Degradation) ### Strategy D: Response Degradation Tiers **Purpose**: Maintain partial functionality when full LLM capabilities unavailable. #### Degradation Tier Framework | Tier | Description | Use Case | User Experience Impact | |------|-------------|----------|------------------------| | **Tier 0 (Full)** | Complete LLM processing | Normal operation | None | | **Tier 1 (Low-Context)** | Return canned/template responses with minimal personalization | Chatbots, support | Medium - Generic but helpful | | **Tier 2 (Partial Processing)** | Process only critical fields/sections of request | Data extraction, summarization | Low-Medium - Incomplete results | | **Tier 3 (Human Handoff Queue)** | Defer to human agents with async notification | Customer service, compliance | Low - Delayed but accurate | #### Implementation Code ```python class DegradationStrategy: TIER_0_FULL = "full_processing" TIER_1_LOW_CONTEXT = "low_context" TIER_2_PARTIAL = "partial_processing" TIER_3_HUMAN_HANDOFF = "human_handoff" def __init__(self): self.current_tier = self.TIER_0_FULL def degrade(self, reason: str) -> str: """Move to next degradation tier""" tiers = [ self.TIER_1_LOW_CONTEXT, self.TIER_2_PARTIAL, self.TIER_3_HUMAN_HANDOFF ] current_idx = tiers.index(self.current_tier) if self.current_tier in tiers else -1 if current_idx < len(tiers) - 1: old_tier = self.current_tier self.current_tier = tiers[current_idx + 1] print(f"⬇️ DEGRADATION: {old_tier} → {self.current_tier}") return self.current_tier else: return self.TIER_3_HUMAN_HANDOFF async def process_request(self, prompt: str) -> dict: """Process request based on current degradation tier""" if self.current_tier == self.TIER_0_FULL: return await self._full_processing(prompt) elif self.current_tier == self.TIER_1_LOW_CONTEXT: # Use templated responses with minimal LLM involvement template = self._find_template(prompt) return { 'response': template['text'], 'tier': 'low_context', 'suggestions': template.get('follow_up_questions', []) } elif self.current_tier == self.TIER_2_PARTIAL: # Process only critical components extracted_intent = await self._quick_intent_detection(prompt) return { 'intent': extracted_intent, 'response': f"Recognized intent: {extracted_intent}. Processing limited due to system constraints.", 'tier': 'partial' } else: # TIER_3_HUMAN_HANDOFF await self._queue_for_human(prompt) return { 'status': 'queued', 'message': 'Your request has been queued for human review.', 'estimated_wait': '< 2 hours' } def _find_template(self, prompt: str) -> dict: """Simple keyword-based template matching""" templates = { 'greeting': {'text': "Hello! How can I assist you today?", 'follow_up_questions': ['Pricing', 'Features']}, 'pricing': {'text': "Our pricing starts at $X/month. Would you like a demo?", 'follow_up_questions': ['Demo', 'Enterprise']}, # ... more templates } for keyword, template in templates.items(): if keyword.lower() in prompt.lower(): return template return {'text': "I'm currently operating with limited functionality. How can I help?", 'follow_up_questions': []} ``` ### Strategy E: Asynchronous Processing Queue **Purpose**: Accept requests during outage, process when services recover. #### Implementation Code ```python import asyncio from typing import Dict, Any from dataclasses import dataclass, field @dataclass class ProcessedRequest: request_id: str prompt: str status: str = "pending" | "processing" | "completed" | "failed" response: Optional[Any] = None created_at: float = field(default_factory=time.time) class AsyncProcessingQueue: def __init__(self, max_pending=1000): self.queue: Dict[str, ProcessedRequest] = {} self.max_pending = max_pending async def enqueue(self, request_data: dict) -> str: """Add request to processing queue""" if len(self.queue) >= self.max_pending: raise QueueFullError("System at capacity. Please try again later.") request_id = f"req_{uuid.uuid4().hex[:12]}" self.queue[request_id] = ProcessedRequest( request_id=request_id, prompt=request_data['prompt'], status="pending" ) # Trigger background processing asyncio.create_task(self._process_queued_request(request_id)) return { 'request_id': request_id, 'status': 'queued', 'message': 'Your request is being processed. Check back in a moment.', 'check_endpoint': f'/api/request/{request_id}/status' } async def _process_queued_request(self, request_id: str): """Background task to process queued requests""" req = self.queue.get(request_id) if not req or req.status != "pending": return req.status = "processing" try: # Wait for LLM availability (with timeout) response = await asyncio.wait_for( self._call_llm_with_retry(req.prompt), timeout=300 # 5 minute max wait ) req.response = response req.status = "completed" except Exception as e: req.status = "failed" print(f"❌ Failed to process {request_id}: {e}") # WebSocket-based live status updates async def websocket_status_checker(request_id: str): """Provide real-time status updates via WebSocket""" while True: req = queue.queue.get(request_id) if not req or req.status in ["completed", "failed"]: await send_to_client({ 'status': req.status, 'response': req.response, 'final': True }) break await send_to_client({'status': req.status}) await asyncio.sleep(2) # Poll every 2 seconds ``` --- ## Level 4: Advanced Recovery Mechanisms ### Strategy F: Model Warm-Up and Health Check **Purpose**: Proactive monitoring prevents unexpected failures. #### Implementation Code ```python import aiohttp from typing import List class LLMHealthMonitor: def __init__(self, providers: List[dict]): self.providers = providers self.health_status: Dict[str, dict] = {} async def check_provider_health(self, provider_config: dict) -> dict: """Check if provider is healthy and responsive""" try: start_time = time.time() # Lightweight health check request response = await aiohttp.ClientSession().post( provider_config['health_endpoint'], json={'text': 'test'}, | 'degraded' return { 'provider': provider_config['name'], 'status': status, 'latency_ms': latency_ms, 'available': available } except Exception as e: return { 'provider': provider_config['name'], 'status': 'unavailable', 'error': str(e), 'available': False } async def get_optimal_provider(self) -> Optional[dict]: """Select best available provider based on health and latency""" healthy_providers = [] for config in self.providers: health = await self.check_provider_health(config) if health['available']: healthy_providers.append({ **config, 'health': health }) # Sort by status priority then latency def sort_key(p): status_priority = {'healthy': 0, 'degraded': 1} return (status_priority.get(p['health']['status'], 2), p['health']['latency_ms']) healthy_providers.sort(key=sort_key) return healthy_providers[0] if healthy_providers else None # Pre-warming strategy for peak hours async def warm_up_models(scheduled_times: List[int]): """Pre-warm models before expected traffic spikes""" for hour in scheduled_times: # e.g., [8, 12, 14] (business hours) current_hour = datetime.now().hour if abs(current_hour - hour) < 30: # Within 30 minutes of peak print(f"🔥 Warming up models for {hour}:00 traffic spike") # Send warm-up requests to all providers tasks = [ aiohttp.ClientSession().post( p['warmup_endpoint'], json={'text': 'System warmup request'} ) for p in providers if p.get('warmup_endpoint') ] await asyncio.gather(*tasks, return_exceptions=True) ``` ### Strategy G: Hybrid Human-in-the-Loop Fallback **Purpose**: Seamless transition to human agents when automated systems exhausted. #### Implementation Code ```python from enum import Enum class HandoffPriority(Enum): CRITICAL = 0 # Immediate human attention required HIGH = 1 # Within 30 minutes NORMAL = 2 # Within 4 hours LOW = 3 # End of day processing class HumanHandoffSystem: def __init__(self, ticket_system_api): self.ticket_api = ticket_system_api def determine_handoff_priority(self, request_context: dict) -> HandoffPriority: """Determine appropriate handoff priority""" # Critical indicators critical_keywords = ['urgent', 'critical', 'emergency', 'outage'] if any(kw in request_context.get('prompt', '').lower() for kw in critical_keywords): return HandoffPriority.CRITICAL # Business value assessment customer_tier = request_context.get('customer_tier', 'standard') if customer_tier == 'enterprise': return HandoffPriority.HIGH return HandoffPriority.NORMAL async def create_handoff_ticket(self, prompt: str, context: dict) -> dict: """Create ticket for human processing""" priority = self.determine_handoff_priority(context) ticket_data = { 'title': f"LLM Fallback - Human Review Required", 'description': f""" Original Request: {prompt} Context: - Customer Tier: {context.get('customer_tier')} - Attempted Models: {context.get('failed_providers', [])} - Timestamp: {datetime.now().isoformat()} """, 'priority': priority.value, 'tags': ['llm-fallback', 'human-review'], 'metadata': context } ticket = await self.ticket_api.create(ticket_data) return { 'ticket_id': ticket['id'], 'priority': priority.name, 'estimated_response_time': self._get_eta(priority), 'tracking_url': f"{self.ticket_api.base_url}/{ticket['id']}" } def _get_eta(self, priority: HandoffPriority) -> str: """Get estimated response time based on priority""" eta_map = { HandoffPriority.CRITICAL: "< 15 minutes", HandoffPriority.HIGH: "< 30 minutes", HandoffPriority.NORMAL: "< 4 hours", HandoffPriority.LOW: "Next business day" } return eta_map[priority] ``` --- ## Integration Framework: Complete Fallback System ### Unified Orchestration Layer **Purpose**: Coordinate all fallback strategies into a single cohesive system. #### Implementation Code ```python from dataclasses import dataclass @dataclass class FallbackConfiguration: """Centralized fallback configuration""" # Caching settings cache_enabled: bool = True cache_ttl_seconds: int = 3600 # Circuit breaker settings circuit_breaker_enabled: bool = True failure_threshold: int = 5 recovery_timeout: int = 120 # Provider hierarchy primary_provider: str = "gpt-4-turbo" fallback_providers: List[str] = field(default_factory=lambda: [ "claude-3-sonnet", "gemini-pro", "ollama-local" ]) # Degradation settings auto_degrade_on_failure: bool = True max_degradation_tier: str = "human_handoff" class ResilientLLMSystem: def __init__(self, config: FallbackConfiguration): self.config = config # Initialize components self.cache_layer = LLMCacheLayer(cache_ttl=config.cache_ttl_seconds) self.circuit_breaker = LLMSafetyCircuit( failure_threshold=config.failure_threshold, timeout_seconds=config.recovery_timeout ) self.provider_router = MultiProviderRouter(self._load_providers()) self.degradation_strategy = DegradationStrategy() async def execute_request_with_fallbacks( self, prompt: str, llm_config: dict ) -> dict | None: """Execute LLM request with comprehensive fallback chain""" # Level 1: Cache check (fastest) if self.config.cache_enabled: cached = await self.cache_layer.get_cached_response(prompt) if cached: return {**cached, 'source': 'cache'} # Level 2: Circuit breaker protection if self.circuit_breaker.state == CircuitState.OPEN: logger.warning("Circuit breaker OPEN - using fallback") # Level 3: Provider routing with fallback try: response = await self.provider_router.call_with_fallback( prompt=prompt, config=llm_config ) # Cache successful response if self.config.cache_enabled and response['provider'] == self.config.primary_provider: self.cache_layer.cache_response(prompt, response) return response except ExhaustedProvidersError as e: logger.error(f"All providers failed: {e}") # Level 4: Degradation strategy current_tier = self.degradation_strategy.current_tier if current_tier != DegradationStrategy.TIER_3_HUMAN_HANDOFF: degraded_response = await self.degradation_strategy.process_request(prompt) return { **degraded_response, 'source': 'degraded', 'tier': current_tier, 'original_prompt': prompt # For potential reprocessing } else: # Level 5: Human handoff (last resort) handoff = await self.handoff_system.create_handoff_ticket( prompt=prompt, context={'failed_providers': [p.provider_name for p in self.providers]} ) return { 'status': 'human_handoff', **handoff, 'message': 'Your request requires human attention. Our team will respond shortly.' } # Deployment configuration fallback_system = ResilientLLMSystem( cache_layer=LLMCacheLayer(cache_ttl=7200), # 2 hour TTL for business context provider_router=MultiProviderRouter(providers), circuit_breaker=LLMSafetyCircuit(failure_threshold=3, timeout_seconds=60), degradation_strategy=DegradationStrategy(), handoff_system=HumanHandoffSystem(ticket_api) ) ``` --- ## Performance Benchmarks and Metrics ### Recommended Monitoring Dashboard | Metric | Target | Alert Threshold | Tool | |--------|--------|-----------------|------| | **Response Time (P95)** | < 2s | > 5s | Prometheus/Datadog | | **Cache Hit Rate** | > 60% | < 40% | Custom | | **Fallback Trigger Rate** | < 5% of requests | > 10% | Sentry | | **Circuit Breaker State** | CLOSED | OPEN/HALF_OPEN | Grafana | | **Degraded Mode Duration** | < 5 min/day | > 30 min | Uptime Robot | | **Human Handoff Volume** | < 1% of traffic | > 2% | Jira/Zendesk | ### Cost Analysis: Fallback Strategy ROI ```python def calculate_fallback_roi(): """Estimate cost savings from fallback implementation""" costs = { 'primary_llm_per_1k_tokens': 0.03, # GPT-4 Turbo ~$0.01/1K input + $0.03/1K output 'fallback_llm_per_1k_tokens': 0.002, # Smaller model/local inference 'cache_hit_savings': 0.97, # Near-zero cost for cache hits 'downtime_cost_per_hour': 5000 # Business impact estimate } monthly_metrics = { 'total_requests': 1_000_000, 'avg_tokens_per_request': 2_000, 'cache_hit_rate': 0.65, 'fallback_triggers': 0.08, # 8% of requests hit fallback 'avoided_downtime_hours': 4 | Monthly avoided downtime } cache_savings = (monthly_metrics['total_requests'] * monthly_metrics['avg_tokens_per_request'] / 1000 * costs['primary_llm_per_1k_tokens'] * monthly_metrics['cache_hit_rate'] * costs['cache_hit_savings']) fallback_savings = (monthly_metrics['total_requests'] * monthly_metrics['fallback_triggers'] * monthly_metrics['avg_tokens_per_request'] / 1000 * (costs['primary_llm_per_1k_tokens'] - costs['fallback_llm_per_1k_tokens'])) downtime_avoided_savings = monthly_metrics['avoided_downtime_hours'] * costs['downtime_cost_per_hour'] return { 'monthly_cache_savings': f"${cache_savings:,.2f}", 'monthly_fallback_savings': f"${fallback_savings:,.2f}", 'monthly_downtime_avoided': f"${downtime_avoided_savings:,.2f}", 'total_monthly_roi': f"${cache_savings + fallback_savings + downtime_avoided_savings:,.2f}" } # Sample output: Monthly savings of $15,000 - $45,000 depending on scale ``` --- ## Key Takeaways and Best Practices ### Implementation Priority Matrix | Phase | Strategy | Complexity | Impact | Timeline | High | Cache Layer + Circuit Breaker | Low | High | Week 1-2 | | 2 (Medium) | Multi-Provider Fallback | Medium | Very High | Week 3-4 | | 3 (High) | Degradation Tiers + Async Queue | High | High | Month 2 | | 4 (Advanced) | Health Monitoring + Auto-Recovery | Medium | Medium | Month 3 | ### Critical Success Factors 1. **Fail Fast, Fail Safe**: Never block user requests; always provide some form of response 2. **Graceful Degradation**: Prefer partial functionality over complete failure 3. **Observability First**: Instrument every fallback layer for debugging 4. **Cost-Aware Design**: Consider cost implications at each fallback tier 5. **Human-in-the-Loop**: Maintain path to human intervention for critical cases ### Common Pitfalls to Avoid ```python COMMON_MISTAKES = { 'sync_fallbacks': "Blocking synchronous fallbacks kill UX - always async", 'no_caching': "Repeating same requests costs 10x without cache layer", 'single_provider': "Single provider dependency = single point of failure", 'silent_failures': "Always log and alert on fallback triggers", 'ignoring_cost': "Fallback to cheaper models can actually INCREASE cost if not monitored" } ``` --- ## Appendix: Reference Implementation ### Complete System Architecture Diagram (Mermaid) ```mermaid graph TB subgraph Client Layer A[Client Request] --> B{Cache Check?} B -->|Hit| C[Return Cached Response] B -->|Miss| D[Circuit Breaker] end subgraph Routing Layer D -->|Closed| E{Primary Provider} D -->|Open| F[Fallback Route] E -->|Success| G[Process Request] E -->|Fail| H{Fallback 1: Alt Provider} H -->|Success| G H -->|Fail| I{Fallback 2: Local Model} end subgraph Degradation Layer I -->|Success| G I -->|Fail| J{Degradation Tier 1} J --> K[Template Response] J --> L{Degradation Tier 2} L --> M[Async Queue] L --> N{Human Handoff} end subgraph Recovery Layer O[(Health Monitor)] -.->|Check| E O -.->|Warm-up| I P[(Background Worker)] -->|Process| M Q[(Human Agent)] -->|Resolve| N end G --> R[Success Response] C --> R K --> S[Degraded Response] M --> T[Queued Notification] N --> U[Handoff Confirmation] ``` ### Error Handling Hierarchy ```python class LLMFallbackError(Exception): """Base exception for all fallback-related errors""" class CacheMiss(LLMFallbackError): """Cache miss - proceed to live request""" class CircuitBreakerOpen(LLMFallbackError): """Circuit breaker is open - skip primary provider""" class ProviderUnavailable(LLMFallbackError): """Specific provider is unavailable""" class ExhaustedProvidersError(LLMFallbackError): """All providers exhausted - trigger degradation""" class DegradationLimitReached(LLMFallbackError): """Maximum degradation tier reached - human handoff required""" ``` --- ## Verification & Testing Recommendations ### Pre-Deployment Checklist ```bash # 1. Unit Tests for Each Component pytest tests/test_cache_layer.py --coverage=95 pytest tests/test_circuit_breaker.py --coverage=90 pytest tests/test_provider_router.py --coverage=85 # 2. Integration Test - Fallback Chain pytest tests/integration/test_fallback_chain.py -v # 3. Chaos Engineering Simulation python scripts/chaos_test.py --simulate-outage --duration=60s # 4. Load Testing with Degradation locust -f locustfiles/fallback_load_test.py --users=1000 --spawn-rate=100 ``` ### Monitoring Validation Queries (PromQL) ```promql # Fallback trigger rate (should be < 5%) rate(llm_fallback_triggers_total[5m]) / rate(llm_requests_total[5m]) # Circuit breaker state change frequency increase(circuit_breaker_state_changes_total[1h]) # Degradation tier distribution histogram_quantile(0.95, llm_response_latency_seconds_bucket) by (tier) # Cache efficiency metrics rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m])) ``` --- ## Final Recommendations Summary ### Quick-Start Implementation Path 1. **Week 1**: Implement caching layer and basic error handling 2. **Week 2**: Add circuit breaker pattern for primary provider 3. **Week 3**: Configure secondary/fallback providers 4. **Week 4**: Deploy degradation tiers for graceful failure 5. **Month 2**: Add health monitoring and auto-recovery mechanisms ### Expected Outcomes | Metric | Before | After (Target) | Improvement | Availability | 98% | 99.9% | +1.9x | | Mean Time To Recovery | 30 min | < 5 min | -83% | | User-Side Errors | ~2% of requests | < 0.1% | -95% | | Monthly Cost Variance | ±40% | ±15% | Stability +62% | --- *Document Version: 1.0 | Last Updated: 2024 | Classification: Technical Architecture*
?
AI

Did this help? Spendplane AI continuously learns from search interactions to refine the Engineering Manual.

Spendplane Meta-Docs / Trace ID: fallback-strategies-for-mission-critical-llm-downtime-de