Modern .NET Patterns - Principal Engineer Deep Dive

CQRS (Command Query Responsibility Segregation)
Rate Limiting & Throttling
Resilience Patterns with Polly
Health Checks & Diagnostics
Distributed Caching
Interview Questions & Answers

1. CQRS (Command Query Responsibility Segregation)

1.1 CQRS Fundamentals

CQRS is an architectural pattern that separates read operations (queries) from write operations (commands), allowing each to be optimized independently.

Core Principles:

// Traditional approach: Single model for reads and writes
public class ProductService
{
    private readonly AppDbContext _context;

    // Both read and write through same model
    public Product GetProduct(int id) => _context.Products.Find(id);
    public void UpdateProduct(Product product) => _context.SaveChanges();
}

// CQRS approach: Separate models and handlers
public interface ICommand { }
public interface IQuery<TResult> { }

public interface ICommandHandler<TCommand> where TCommand : ICommand
{
    Task HandleAsync(TCommand command, CancellationToken ct = default);
}

public interface IQueryHandler<TQuery, TResult> where TQuery : IQuery<TResult>
{
    Task<TResult> HandleAsync(TQuery query, CancellationToken ct = default);
}

[INTERNALS] Why Separate Reads from Writes:

Aspect	Commands (Write)	Queries (Read)
Optimization	Consistency, validation	Speed, caching
Scaling	Write replicas	Read replicas
Model	Domain-rich, behavior	DTO-flat, view-specific
Caching	Invalidation on change	Aggressive caching
Database	Normalized	Denormalized/materialized

1.2 Simple CQRS Without Event Sourcing

[CODE] Basic CQRS Implementation:

// Commands - express intent to change state
public record CreateProductCommand(
    string Name,
    decimal Price,
    int CategoryId) : ICommand;

public record UpdateProductPriceCommand(
    int ProductId,
    decimal NewPrice) : ICommand;

// Queries - express intent to retrieve data
public record GetProductByIdQuery(int ProductId) : IQuery<ProductDto>;

public record GetProductsByCategoryQuery(
    int CategoryId,
    int Page = 1,
    int PageSize = 20) : IQuery<PagedResult<ProductDto>>;

// DTOs - optimized for reading
public record ProductDto(
    int Id,
    string Name,
    decimal Price,
    string CategoryName,
    bool InStock);

public record PagedResult<T>(
    IReadOnlyList<T> Items,
    int TotalCount,
    int Page,
    int PageSize)
{
    public int TotalPages => (int)Math.Ceiling(TotalCount / (double)PageSize);
    public bool HasNextPage => Page < TotalPages;
    public bool HasPreviousPage => Page > 1;
}

[CODE] Command Handler with Validation:

public class CreateProductCommandHandler : ICommandHandler<CreateProductCommand>
{
    private readonly WriteDbContext _writeContext;
    private readonly IValidator<CreateProductCommand> _validator;
    private readonly IEventPublisher _events;
    private readonly ILogger<CreateProductCommandHandler> _logger;

    public CreateProductCommandHandler(
        WriteDbContext writeContext,
        IValidator<CreateProductCommand> validator,
        IEventPublisher events,
        ILogger<CreateProductCommandHandler> logger)
    {
        _writeContext = writeContext;
        _validator = validator;
        _events = events;
        _logger = logger;
    }

    public async Task HandleAsync(CreateProductCommand command, CancellationToken ct)
    {
        // 1. Validate
        var validationResult = await _validator.ValidateAsync(command, ct);
        if (!validationResult.IsValid)
        {
            throw new ValidationException(validationResult.Errors);
        }

        // 2. Check business rules
        var categoryExists = await _writeContext.Categories
            .AnyAsync(c => c.Id == command.CategoryId, ct);

        if (!categoryExists)
        {
            throw new BusinessRuleException($"Category {command.CategoryId} not found");
        }

        // 3. Create domain entity
        var product = new Product
        {
            Name = command.Name,
            Price = command.Price,
            CategoryId = command.CategoryId,
            CreatedAt = DateTime.UtcNow
        };

        // 4. Persist
        _writeContext.Products.Add(product);
        await _writeContext.SaveChangesAsync(ct);

        _logger.LogInformation("Product {ProductId} created: {Name}", product.Id, product.Name);

        // 5. Publish event for read model update
        await _events.PublishAsync(new ProductCreatedEvent(product.Id, product.Name), ct);
    }
}

// FluentValidation validator
public class CreateProductCommandValidator : AbstractValidator<CreateProductCommand>
{
    public CreateProductCommandValidator()
    {
        RuleFor(x => x.Name)
            .NotEmpty()
            .MaximumLength(200);

        RuleFor(x => x.Price)
            .GreaterThan(0)
            .LessThan(1_000_000);

        RuleFor(x => x.CategoryId)
            .GreaterThan(0);
    }
}

[CODE] Query Handler with Projection:

public class GetProductByIdQueryHandler : IQueryHandler<GetProductByIdQuery, ProductDto>
{
    private readonly ReadDbContext _readContext;
    private readonly IDistributedCache _cache;
    private readonly ILogger<GetProductByIdQueryHandler> _logger;

    public GetProductByIdQueryHandler(
        ReadDbContext readContext,
        IDistributedCache cache,
        ILogger<GetProductByIdQueryHandler> logger)
    {
        _readContext = readContext;
        _cache = cache;
        _logger = logger;
    }

    public async Task<ProductDto> HandleAsync(GetProductByIdQuery query, CancellationToken ct)
    {
        // Try cache first
        var cacheKey = $"product:{query.ProductId}";
        var cached = await _cache.GetStringAsync(cacheKey, ct);

        if (cached != null)
        {
            _logger.LogDebug("Cache hit for product {ProductId}", query.ProductId);
            return JsonSerializer.Deserialize<ProductDto>(cached)!;
        }

        // Query with projection (no tracking, optimized SQL)
        var product = await _readContext.Products
            .AsNoTracking()
            .Where(p => p.Id == query.ProductId)
            .Select(p => new ProductDto(
                p.Id,
                p.Name,
                p.Price,
                p.Category.Name,
                p.Stock > 0))
            .FirstOrDefaultAsync(ct);

        if (product == null)
        {
            throw new NotFoundException($"Product {query.ProductId} not found");
        }

        // Cache for future requests
        await _cache.SetStringAsync(
            cacheKey,
            JsonSerializer.Serialize(product),
            new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5)
            },
            ct);

        return product;
    }
}

1.3 CQRS with MediatR

[CODE] MediatR-based Implementation:

// Install: MediatR, MediatR.Extensions.Microsoft.DependencyInjection

// Commands with MediatR
public record CreateProductCommand(
    string Name,
    decimal Price,
    int CategoryId) : IRequest<int>; // Returns product ID

public record UpdateProductPriceCommand(
    int ProductId,
    decimal NewPrice) : IRequest<Unit>;

// Queries with MediatR
public record GetProductByIdQuery(int ProductId) : IRequest<ProductDto>;

// Handler
public class CreateProductHandler : IRequestHandler<CreateProductCommand, int>
{
    private readonly WriteDbContext _context;

    public CreateProductHandler(WriteDbContext context)
    {
        _context = context;
    }

    public async Task<int> Handle(CreateProductCommand request, CancellationToken ct)
    {
        var product = new Product
        {
            Name = request.Name,
            Price = request.Price,
            CategoryId = request.CategoryId
        };

        _context.Products.Add(product);
        await _context.SaveChangesAsync(ct);

        return product.Id;
    }
}

// Pipeline behavior for cross-cutting concerns
public class ValidationBehavior<TRequest, TResponse>
    : IPipelineBehavior<TRequest, TResponse>
    where TRequest : IRequest<TResponse>
{
    private readonly IEnumerable<IValidator<TRequest>> _validators;

    public ValidationBehavior(IEnumerable<IValidator<TRequest>> validators)
    {
        _validators = validators;
    }

    public async Task<TResponse> Handle(
        TRequest request,
        RequestHandlerDelegate<TResponse> next,
        CancellationToken ct)
    {
        if (_validators.Any())
        {
            var context = new ValidationContext<TRequest>(request);

            var validationResults = await Task.WhenAll(
                _validators.Select(v => v.ValidateAsync(context, ct)));

            var failures = validationResults
                .SelectMany(r => r.Errors)
                .Where(f => f != null)
                .ToList();

            if (failures.Count > 0)
            {
                throw new ValidationException(failures);
            }
        }

        return await next();
    }
}

// Logging behavior
public class LoggingBehavior<TRequest, TResponse>
    : IPipelineBehavior<TRequest, TResponse>
    where TRequest : IRequest<TResponse>
{
    private readonly ILogger<LoggingBehavior<TRequest, TResponse>> _logger;

    public LoggingBehavior(ILogger<LoggingBehavior<TRequest, TResponse>> logger)
    {
        _logger = logger;
    }

    public async Task<TResponse> Handle(
        TRequest request,
        RequestHandlerDelegate<TResponse> next,
        CancellationToken ct)
    {
        var requestName = typeof(TRequest).Name;

        _logger.LogInformation("Handling {RequestName}: {@Request}", requestName, request);

        var sw = Stopwatch.StartNew();
        var response = await next();
        sw.Stop();

        _logger.LogInformation(
            "Handled {RequestName} in {ElapsedMs}ms",
            requestName,
            sw.ElapsedMilliseconds);

        return response;
    }
}

// DI Registration
public static class CqrsServiceExtensions
{
    public static IServiceCollection AddCqrs(this IServiceCollection services)
    {
        services.AddMediatR(cfg =>
        {
            cfg.RegisterServicesFromAssemblyContaining<CreateProductHandler>();
            cfg.AddBehavior(typeof(IPipelineBehavior<,>), typeof(ValidationBehavior<,>));
            cfg.AddBehavior(typeof(IPipelineBehavior<,>), typeof(LoggingBehavior<,>));
        });

        services.AddValidatorsFromAssemblyContaining<CreateProductCommandValidator>();

        return services;
    }
}

// Controller usage
[ApiController]
[Route("api/[controller]")]
public class ProductsController : ControllerBase
{
    private readonly IMediator _mediator;

    public ProductsController(IMediator mediator)
    {
        _mediator = mediator;
    }

    [HttpPost]
    public async Task<ActionResult<int>> Create(CreateProductCommand command)
    {
        var productId = await _mediator.Send(command);
        return CreatedAtAction(nameof(GetById), new { id = productId }, productId);
    }

    [HttpGet("{id}")]
    public async Task<ActionResult<ProductDto>> GetById(int id)
    {
        var product = await _mediator.Send(new GetProductByIdQuery(id));
        return Ok(product);
    }
}

1.4 Separate Read/Write Databases

[PRODUCTION] CQRS with Database Separation:

// Separate DbContexts for read and write
public class WriteDbContext : DbContext
{
    public WriteDbContext(DbContextOptions<WriteDbContext> options) : base(options) { }

    public DbSet<Product> Products => Set<Product>();
    public DbSet<Category> Categories => Set<Category>();
    public DbSet<Order> Orders => Set<Order>();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Full domain model with relationships, constraints
        modelBuilder.Entity<Product>(entity =>
        {
            entity.HasKey(e => e.Id);
            entity.Property(e => e.Name).IsRequired().HasMaxLength(200);
            entity.Property(e => e.Price).HasPrecision(18, 2);
            entity.HasOne(e => e.Category)
                  .WithMany(c => c.Products)
                  .HasForeignKey(e => e.CategoryId);
            entity.Property(e => e.RowVersion).IsRowVersion();
        });
    }
}

public class ReadDbContext : DbContext
{
    public ReadDbContext(DbContextOptions<ReadDbContext> options) : base(options) { }

    public DbSet<ProductReadModel> Products => Set<ProductReadModel>();
    public DbSet<OrderSummaryReadModel> OrderSummaries => Set<OrderSummaryReadModel>();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Denormalized read models, optimized for queries
        modelBuilder.Entity<ProductReadModel>(entity =>
        {
            entity.HasKey(e => e.Id);
            entity.ToTable("ProductReadModels");
            // No foreign keys - flat structure
        });
    }
}

// Read model - denormalized for fast queries
public class ProductReadModel
{
    public int Id { get; set; }
    public string Name { get; set; } = "";
    public decimal Price { get; set; }
    public string CategoryName { get; set; } = "";
    public int StockQuantity { get; set; }
    public DateTime LastUpdated { get; set; }

    // Calculated/cached fields
    public bool InStock => StockQuantity > 0;
    public string PriceDisplay => Price.ToString("C");
}

// Sync read models via events
public class ProductCreatedEventHandler : INotificationHandler<ProductCreatedEvent>
{
    private readonly ReadDbContext _readContext;
    private readonly WriteDbContext _writeContext;

    public ProductCreatedEventHandler(
        ReadDbContext readContext,
        WriteDbContext writeContext)
    {
        _readContext = readContext;
        _writeContext = writeContext;
    }

    public async Task Handle(ProductCreatedEvent notification, CancellationToken ct)
    {
        // Fetch from write DB
        var product = await _writeContext.Products
            .Include(p => p.Category)
            .FirstOrDefaultAsync(p => p.Id == notification.ProductId, ct);

        if (product == null) return;

        // Create read model
        var readModel = new ProductReadModel
        {
            Id = product.Id,
            Name = product.Name,
            Price = product.Price,
            CategoryName = product.Category?.Name ?? "Uncategorized",
            StockQuantity = product.Stock,
            LastUpdated = DateTime.UtcNow
        };

        _readContext.Products.Add(readModel);
        await _readContext.SaveChangesAsync(ct);
    }
}

// DI Setup for multiple databases
public static void ConfigureDatabases(IServiceCollection services, IConfiguration config)
{
    // Write database - primary with full model
    services.AddDbContext<WriteDbContext>(options =>
        options.UseSqlServer(config.GetConnectionString("WriteDb")));

    // Read database - can be replica or separate optimized DB
    services.AddDbContext<ReadDbContext>(options =>
        options.UseSqlServer(config.GetConnectionString("ReadDb"))
               .UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking));
}

[BENCHMARK] CQRS Performance Comparison:

Scenario	Traditional	CQRS	Improvement
Simple read	5ms	3ms	40%
Complex join read	50ms	8ms	84%
Write with validation	20ms	22ms	-10% (overhead)
High read/write ratio (100:1)	100ms avg	35ms avg	65%
Cache-friendly reads	5ms	0.5ms	90%

2. Rate Limiting & Throttling

2.1 Built-in Rate Limiting Middleware (.NET 7+)

.NET 7 introduced native rate limiting with several algorithms:

[CODE] Basic Rate Limiting Setup:

// Program.cs
var builder = WebApplication.CreateBuilder(args);

// Add rate limiting services
builder.Services.AddRateLimiter(options =>
{
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;

    // Global rate limit
    options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(context =>
        RateLimitPartition.GetFixedWindowLimiter(
            partitionKey: context.Connection.RemoteIpAddress?.ToString() ?? "unknown",
            factory: _ => new FixedWindowRateLimiterOptions
            {
                PermitLimit = 100,
                Window = TimeSpan.FromMinutes(1),
                QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
                QueueLimit = 10
            }));

    // Named policy for specific endpoints
    options.AddFixedWindowLimiter("api", options =>
    {
        options.PermitLimit = 50;
        options.Window = TimeSpan.FromMinutes(1);
        options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
        options.QueueLimit = 5;
    });

    // Sliding window for smoother limiting
    options.AddSlidingWindowLimiter("sliding", options =>
    {
        options.PermitLimit = 100;
        options.Window = TimeSpan.FromMinutes(1);
        options.SegmentsPerWindow = 6; // 10-second segments
        options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
        options.QueueLimit = 10;
    });

    // Token bucket for burst handling
    options.AddTokenBucketLimiter("burst", options =>
    {
        options.TokenLimit = 100;
        options.ReplenishmentPeriod = TimeSpan.FromSeconds(10);
        options.TokensPerPeriod = 20;
        options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
        options.QueueLimit = 10;
        options.AutoReplenishment = true;
    });

    // Concurrency limiter
    options.AddConcurrencyLimiter("concurrent", options =>
    {
        options.PermitLimit = 10;
        options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
        options.QueueLimit = 5;
    });

    // Custom rejection response
    options.OnRejected = async (context, ct) =>
    {
        context.HttpContext.Response.StatusCode = 429;
        context.HttpContext.Response.ContentType = "application/json";

        var retryAfter = context.Lease.TryGetMetadata(
            MetadataName.RetryAfter, out var retryAfterValue)
            ? retryAfterValue.TotalSeconds
            : 60;

        context.HttpContext.Response.Headers.RetryAfter = retryAfter.ToString("0");

        await context.HttpContext.Response.WriteAsJsonAsync(new
        {
            error = "Too many requests",
            retryAfterSeconds = retryAfter,
            message = "Rate limit exceeded. Please try again later."
        }, ct);
    };
});

var app = builder.Build();

// Apply rate limiting middleware (before other middleware)
app.UseRateLimiter();

app.MapGet("/api/products", () => Results.Ok())
    .RequireRateLimiting("api");

app.MapGet("/api/search", () => Results.Ok())
    .RequireRateLimiting("sliding");

app.MapPost("/api/upload", () => Results.Ok())
    .RequireRateLimiting("concurrent");

app.Run();

2.2 Rate Limiting Algorithms Explained

[INTERNALS] How Each Algorithm Works:

Fixed Window:
┌─────────────┐┌─────────────┐┌─────────────┐
│   Window 1  ││   Window 2  ││   Window 3  │
│ Limit: 100  ││ Limit: 100  ││ Limit: 100  │
│ Used: 85    ││ Used: 0     ││ Used: 0     │
└─────────────┘└─────────────┘└─────────────┘
     00:00         01:00         02:00
Problem: Burst at window boundary (100 requests at 0:59 + 100 at 1:00)

Sliding Window (6 segments):
┌──┬──┬──┬──┬──┬──┐
│15│20│25│10│18│12│ = 100 total (current window)
└──┴──┴──┴──┴──┴──┘
  ↑                ↑
Old segment drops  New segment added
Smoother: Always considers last N time units

Token Bucket:
┌─────────────────────┐
│ Bucket (max 100)    │
│ Current: 75 tokens  │
│ +20 every 10 sec    │
└─────────────────────┘
Allows bursts up to bucket size, then throttles

Concurrency:
┌─────────────────────┐
│ Active: 8/10        │
│ Queue: 3/5          │
└─────────────────────┘
Limits simultaneous requests, not rate

[CODE] Algorithm Comparison:

public class RateLimitAlgorithmComparison
{
    public static void ConfigureAllAlgorithms(RateLimiterOptions options)
    {
        // Fixed Window: Best for simple scenarios
        // - Predictable reset times
        // - Potential for boundary bursts
        options.AddFixedWindowLimiter("fixed", opt =>
        {
            opt.PermitLimit = 100;
            opt.Window = TimeSpan.FromMinutes(1);
        });

        // Sliding Window: Best for smooth rate limiting
        // - No boundary bursts
        // - Higher memory usage (tracks segments)
        options.AddSlidingWindowLimiter("sliding", opt =>
        {
            opt.PermitLimit = 100;
            opt.Window = TimeSpan.FromMinutes(1);
            opt.SegmentsPerWindow = 6; // More segments = smoother but more memory
        });

        // Token Bucket: Best for allowing controlled bursts
        // - Good for APIs that need burst capacity
        // - Tokens replenish continuously
        options.AddTokenBucketLimiter("token", opt =>
        {
            opt.TokenLimit = 100;          // Max burst size
            opt.ReplenishmentPeriod = TimeSpan.FromSeconds(1);
            opt.TokensPerPeriod = 10;      // 10/sec = 600/min sustained rate
            opt.AutoReplenishment = true;
        });

        // Concurrency: Best for resource protection
        // - Limits parallel execution, not rate
        // - Good for expensive operations
        options.AddConcurrencyLimiter("concurrency", opt =>
        {
            opt.PermitLimit = 10;          // Max 10 concurrent
            opt.QueueLimit = 20;           // 20 can wait in queue
        });
    }
}

2.3 Per-User and Per-Endpoint Rate Limiting

[CODE] Advanced Partitioning:

builder.Services.AddRateLimiter(options =>
{
    // Per-user rate limiting (authenticated users)
    options.AddPolicy("per-user", context =>
    {
        var userId = context.User?.FindFirstValue(ClaimTypes.NameIdentifier);

        if (userId != null)
        {
            // Authenticated users get higher limits
            return RateLimitPartition.GetTokenBucketLimiter(userId, _ =>
                new TokenBucketRateLimiterOptions
                {
                    TokenLimit = 1000,
                    ReplenishmentPeriod = TimeSpan.FromMinutes(1),
                    TokensPerPeriod = 100,
                    AutoReplenishment = true
                });
        }
        else
        {
            // Anonymous users get lower limits by IP
            var ip = context.Connection.RemoteIpAddress?.ToString() ?? "unknown";
            return RateLimitPartition.GetTokenBucketLimiter(ip, _ =>
                new TokenBucketRateLimiterOptions
                {
                    TokenLimit = 100,
                    ReplenishmentPeriod = TimeSpan.FromMinutes(1),
                    TokensPerPeriod = 10,
                    AutoReplenishment = true
                });
        }
    });

    // Tiered rate limiting by subscription
    options.AddPolicy("tiered", context =>
    {
        var tier = context.User?.FindFirstValue("subscription_tier") ?? "free";

        var limits = tier switch
        {
            "enterprise" => (1000, 200),
            "pro" => (500, 100),
            "basic" => (100, 20),
            _ => (50, 10) // free
        };

        return RateLimitPartition.GetTokenBucketLimiter(
            context.User?.Identity?.Name ?? "anonymous",
            _ => new TokenBucketRateLimiterOptions
            {
                TokenLimit = limits.Item1,
                ReplenishmentPeriod = TimeSpan.FromMinutes(1),
                TokensPerPeriod = limits.Item2,
                AutoReplenishment = true
            });
    });

    // Endpoint-specific with cost weighting
    options.AddPolicy("weighted", context =>
    {
        var endpoint = context.GetEndpoint()?.DisplayName ?? "unknown";
        var userId = context.User?.FindFirstValue(ClaimTypes.NameIdentifier) ?? "anon";

        // Different endpoints have different "costs"
        var cost = endpoint switch
        {
            var e when e.Contains("search") => 5,  // Expensive operation
            var e when e.Contains("export") => 20, // Very expensive
            _ => 1 // Normal
        };

        return RateLimitPartition.GetTokenBucketLimiter(
            $"{userId}:{endpoint}",
            _ => new TokenBucketRateLimiterOptions
            {
                TokenLimit = 100 / cost,
                ReplenishmentPeriod = TimeSpan.FromMinutes(1),
                TokensPerPeriod = 10 / cost,
                AutoReplenishment = true
            });
    });
});

2.4 Distributed Rate Limiting with Redis

[PRODUCTION] Redis-backed Rate Limiting:

// Custom distributed rate limiter using Redis
public class RedisRateLimiter : RateLimiter
{
    private readonly IDatabase _redis;
    private readonly string _keyPrefix;
    private readonly int _permitLimit;
    private readonly TimeSpan _window;

    public RedisRateLimiter(
        IConnectionMultiplexer redis,
        string keyPrefix,
        int permitLimit,
        TimeSpan window)
    {
        _redis = redis.GetDatabase();
        _keyPrefix = keyPrefix;
        _permitLimit = permitLimit;
        _window = window;
    }

    public override RateLimiterStatistics? GetStatistics() => null;

    protected override RateLimitLease AttemptAcquireCore(int permitCount)
    {
        // Synchronous version - should use async in real implementation
        return AcquireAsync(permitCount, CancellationToken.None)
            .GetAwaiter()
            .GetResult();
    }

    protected override async ValueTask<RateLimitLease> AcquireAsyncCore(
        int permitCount,
        CancellationToken ct)
    {
        var key = _keyPrefix;
        var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
        var windowStart = now - (long)_window.TotalSeconds;

        // Lua script for atomic sliding window
        var script = @"
            local key = KEYS[1]
            local windowStart = tonumber(ARGV[1])
            local now = tonumber(ARGV[2])
            local limit = tonumber(ARGV[3])
            local permitCount = tonumber(ARGV[4])
            local windowSize = tonumber(ARGV[5])

            -- Remove old entries
            redis.call('ZREMRANGEBYSCORE', key, '-inf', windowStart)

            -- Count current requests
            local current = redis.call('ZCARD', key)

            if current + permitCount <= limit then
                -- Add new request(s)
                for i = 1, permitCount do
                    redis.call('ZADD', key, now, now .. ':' .. i .. ':' .. math.random())
                end
                redis.call('EXPIRE', key, windowSize)
                return current + permitCount
            else
                return -1
            end
        ";

        var result = await _redis.ScriptEvaluateAsync(
            script,
            new RedisKey[] { key },
            new RedisValue[] { windowStart, now, _permitLimit, permitCount, (long)_window.TotalSeconds });

        var count = (long)result;

        if (count >= 0)
        {
            return new RedisRateLimitLease(true, count, _permitLimit);
        }

        return new RedisRateLimitLease(false, _permitLimit, _permitLimit);
    }

    public override TimeSpan? IdleDuration => null;

    private class RedisRateLimitLease : RateLimitLease
    {
        private readonly long _currentCount;
        private readonly int _limit;

        public RedisRateLimitLease(bool isAcquired, long currentCount, int limit)
        {
            IsAcquired = isAcquired;
            _currentCount = currentCount;
            _limit = limit;
        }

        public override bool IsAcquired { get; }

        public override IEnumerable<string> MetadataNames =>
            new[] { "CURRENT_COUNT", "LIMIT" };

        public override bool TryGetMetadata(string metadataName, out object? metadata)
        {
            metadata = metadataName switch
            {
                "CURRENT_COUNT" => _currentCount,
                "LIMIT" => _limit,
                _ => null
            };
            return metadata != null;
        }
    }
}

// Usage with DI
public static class RedisRateLimiterExtensions
{
    public static IServiceCollection AddRedisRateLimiting(
        this IServiceCollection services,
        IConfiguration config)
    {
        services.AddSingleton<IConnectionMultiplexer>(
            ConnectionMultiplexer.Connect(config.GetConnectionString("Redis")!));

        services.AddRateLimiter(options =>
        {
            options.AddPolicy("redis-global", context =>
            {
                var redis = context.RequestServices
                    .GetRequiredService<IConnectionMultiplexer>();

                var ip = context.Connection.RemoteIpAddress?.ToString() ?? "unknown";

                return RateLimitPartition.Get($"global:{ip}", _ =>
                    new RedisRateLimiter(redis, $"ratelimit:{ip}", 100, TimeSpan.FromMinutes(1)));
            });
        });

        return services;
    }
}

3. Resilience Patterns with Polly

3.1 Retry Strategies

[CODE] Basic Retry Patterns:

// Install: Microsoft.Extensions.Http.Resilience (includes Polly v8)
// Or: Polly, Polly.Extensions.Http

// Immediate retry
var immediateRetryPolicy = Policy
    .Handle<HttpRequestException>()
    .RetryAsync(3, onRetry: (exception, retryCount) =>
    {
        Console.WriteLine($"Retry {retryCount} due to: {exception.Message}");
    });

// Exponential backoff with jitter
var exponentialRetryPolicy = Policy
    .Handle<HttpRequestException>()
    .Or<TimeoutException>()
    .WaitAndRetryAsync(
        retryCount: 5,
        sleepDurationProvider: (retryAttempt, context) =>
        {
            // Exponential: 1s, 2s, 4s, 8s, 16s + jitter
            var exponentialBackoff = TimeSpan.FromSeconds(Math.Pow(2, retryAttempt - 1));
            var jitter = TimeSpan.FromMilliseconds(Random.Shared.Next(0, 1000));
            return exponentialBackoff + jitter;
        },
        onRetryAsync: async (exception, timespan, retryCount, context) =>
        {
            Console.WriteLine($"Retry {retryCount} after {timespan.TotalSeconds}s due to: {exception.Message}");
            await Task.CompletedTask;
        });

// Retry with specific HTTP status codes
var httpRetryPolicy = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(r => r.StatusCode == HttpStatusCode.TooManyRequests)
    .OrResult(r => r.StatusCode >= HttpStatusCode.InternalServerError)
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: (retryCount, response, context) =>
        {
            // Respect Retry-After header if present
            if (response.Result?.Headers.RetryAfter?.Delta.HasValue == true)
            {
                return response.Result.Headers.RetryAfter.Delta.Value;
            }

            return TimeSpan.FromSeconds(Math.Pow(2, retryCount));
        },
        onRetryAsync: (response, timespan, retryCount, context) => Task.CompletedTask);

3.2 Circuit Breaker Pattern

[INTERNALS] Circuit Breaker State Machine:

    ┌─────────────────────────────────────────────────┐
    │                                                 │
    ▼                                                 │
┌────────┐   Failure threshold   ┌────────┐          │
│ CLOSED │ ──────────────────► │  OPEN  │          │
│        │       exceeded        │        │          │
└────────┘                       └────────┘          │
    ▲                                │               │
    │                                │ Break         │
    │                                │ duration      │
    │         Success                ▼               │
    │      ┌───────────────────┬────────────┐       │
    └──────│    HALF-OPEN      │   Failure  │───────┘
           └───────────────────┴────────────┘

[CODE] Circuit Breaker Implementation:

// Basic circuit breaker
var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .Or<TimeoutException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (exception, duration) =>
        {
            Console.WriteLine($"Circuit OPEN for {duration.TotalSeconds}s: {exception.Message}");
        },
        onReset: () =>
        {
            Console.WriteLine("Circuit CLOSED - normal operation resumed");
        },
        onHalfOpen: () =>
        {
            Console.WriteLine("Circuit HALF-OPEN - testing...");
        });

// Advanced circuit breaker with sampling
var advancedCircuitBreaker = Policy
    .Handle<Exception>()
    .AdvancedCircuitBreakerAsync(
        failureThreshold: 0.5,  // 50% failure rate
        samplingDuration: TimeSpan.FromSeconds(30),
        minimumThroughput: 10,  // Need at least 10 requests to evaluate
        durationOfBreak: TimeSpan.FromMinutes(1),
        onBreak: (exception, state, duration, context) =>
        {
            Console.WriteLine($"Circuit broken! State before: {state}, Duration: {duration}");
        },
        onReset: context =>
        {
            Console.WriteLine("Circuit reset - back to normal");
        },
        onHalfOpen: () =>
        {
            Console.WriteLine("Testing if service recovered...");
        });

// For HTTP responses
var httpCircuitBreaker = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(r => (int)r.StatusCode >= 500)
    .AdvancedCircuitBreakerAsync(
        failureThreshold: 0.25,  // 25% failure rate
        samplingDuration: TimeSpan.FromSeconds(60),
        minimumThroughput: 20,
        durationOfBreak: TimeSpan.FromSeconds(30));

// Checking circuit state
public class ServiceClient
{
    private readonly AsyncCircuitBreakerPolicy _circuitBreaker;

    public ServiceClient()
    {
        _circuitBreaker = Policy
            .Handle<Exception>()
            .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));
    }

    public bool IsCircuitOpen => _circuitBreaker.CircuitState == CircuitState.Open;
    public bool IsCircuitHalfOpen => _circuitBreaker.CircuitState == CircuitState.HalfOpen;

    public async Task<string> CallServiceAsync()
    {
        if (IsCircuitOpen)
        {
            // Fast fail without trying
            throw new ServiceUnavailableException("Service is currently unavailable");
        }

        return await _circuitBreaker.ExecuteAsync(async () =>
        {
            // Actual service call
            return await DoServiceCallAsync();
        });
    }

    private Task<string> DoServiceCallAsync() => Task.FromResult("result");
}

3.3 Combined Resilience Pipelines

[CODE] Polly v8 Resilience Pipeline:

// .NET 8 with Microsoft.Extensions.Http.Resilience
var builder = WebApplication.CreateBuilder(args);

// Option 1: Standard resilience handler (recommended)
builder.Services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler();

// Option 2: Custom resilience pipeline
builder.Services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddResilienceHandler("custom", builder =>
    {
        // Rate limiter (innermost - executes first)
        builder.AddRateLimiter(new SlidingWindowRateLimiter(
            new SlidingWindowRateLimiterOptions
            {
                PermitLimit = 100,
                Window = TimeSpan.FromSeconds(30),
                SegmentsPerWindow = 3
            }));

        // Timeout (per-try)
        builder.AddTimeout(TimeSpan.FromSeconds(10));

        // Retry
        builder.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromSeconds(1),
            UseJitter = true,
            BackoffType = DelayBackoffType.Exponential,
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                .Handle<HttpRequestException>()
                .HandleResult(r => r.StatusCode >= HttpStatusCode.InternalServerError)
        });

        // Circuit breaker
        builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            SamplingDuration = TimeSpan.FromSeconds(30),
            MinimumThroughput = 10,
            BreakDuration = TimeSpan.FromSeconds(30),
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                .Handle<HttpRequestException>()
                .HandleResult(r => r.StatusCode >= HttpStatusCode.InternalServerError)
        });

        // Total timeout (outermost)
        builder.AddTimeout(TimeSpan.FromSeconds(60));
    });

// Option 3: Polly v7 style with PolicyWrap
public static class ResiliencePolicies
{
    public static IAsyncPolicy<HttpResponseMessage> GetCombinedPolicy()
    {
        // Policies execute from innermost to outermost
        // Order: Timeout(per-try) -> Retry -> CircuitBreaker -> Timeout(total)

        var timeoutPerTry = Policy
            .TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(10));

        var retry = Policy<HttpResponseMessage>
            .Handle<HttpRequestException>()
            .Or<TimeoutRejectedException>()
            .OrResult(r => (int)r.StatusCode >= 500)
            .WaitAndRetryAsync(3, attempt =>
                TimeSpan.FromSeconds(Math.Pow(2, attempt)) +
                TimeSpan.FromMilliseconds(Random.Shared.Next(0, 1000)));

        var circuitBreaker = Policy<HttpResponseMessage>
            .Handle<HttpRequestException>()
            .Or<TimeoutRejectedException>()
            .OrResult(r => (int)r.StatusCode >= 500)
            .AdvancedCircuitBreakerAsync(0.5, TimeSpan.FromSeconds(30), 10, TimeSpan.FromSeconds(30));

        var totalTimeout = Policy
            .TimeoutAsync<HttpResponseMessage>(TimeSpan.FromMinutes(1));

        // Wrap: outer -> inner execution order
        return Policy.WrapAsync(totalTimeout, circuitBreaker, retry, timeoutPerTry);
    }
}

// Using the combined policy
public class ResilientApiClient
{
    private readonly HttpClient _httpClient;
    private readonly IAsyncPolicy<HttpResponseMessage> _policy;

    public ResilientApiClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
        _policy = ResiliencePolicies.GetCombinedPolicy();
    }

    public async Task<T> GetAsync<T>(string url)
    {
        var response = await _policy.ExecuteAsync(() =>
            _httpClient.GetAsync(url));

        response.EnsureSuccessStatusCode();
        return await response.Content.ReadFromJsonAsync<T>()
            ?? throw new InvalidOperationException("Response was null");
    }
}

3.4 Fallback and Hedging

[CODE] Fallback Strategies:

// Simple fallback
var fallbackPolicy = Policy<string>
    .Handle<HttpRequestException>()
    .Or<TimeoutException>()
    .FallbackAsync(
        fallbackValue: "Default value",
        onFallbackAsync: (exception, context) =>
        {
            Console.WriteLine($"Falling back due to: {exception.Exception?.Message}");
            return Task.CompletedTask;
        });

// Fallback with factory
var factoryFallbackPolicy = Policy<ProductDto>
    .Handle<Exception>()
    .FallbackAsync(
        fallbackAction: async (context, ct) =>
        {
            // Try to get from cache
            var cached = await GetFromCacheAsync(context["productId"]?.ToString());
            if (cached != null) return cached;

            // Return a placeholder
            return new ProductDto(0, "Product Unavailable", 0, "Unknown", false);
        },
        onFallbackAsync: (result, context) =>
        {
            Console.WriteLine($"Using fallback for product {context["productId"]}");
            return Task.CompletedTask;
        });

// Hedging (parallel requests) - Polly v8
builder.Services.AddHttpClient<IApiClient, ApiClient>()
    .AddResilienceHandler("hedging", builder =>
    {
        builder.AddHedging(new HedgingStrategyOptions<HttpResponseMessage>
        {
            MaxHedgedAttempts = 2,
            Delay = TimeSpan.FromMilliseconds(200), // Start second request after 200ms
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                .Handle<HttpRequestException>()
                .HandleResult(r => !r.IsSuccessStatusCode)
        });
    });

// Custom hedging implementation
public class HedgingClient
{
    private readonly HttpClient[] _httpClients;

    public HedgingClient(IEnumerable<HttpClient> clients)
    {
        _httpClients = clients.ToArray();
    }

    public async Task<T> GetWithHedging<T>(string path, CancellationToken ct = default)
    {
        using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);

        // Start all requests
        var tasks = _httpClients
            .Select(client => GetAsync<T>(client, path, cts.Token))
            .ToList();

        // Return first successful response
        while (tasks.Count > 0)
        {
            var completed = await Task.WhenAny(tasks);
            tasks.Remove(completed);

            try
            {
                var result = await completed;
                cts.Cancel(); // Cancel remaining requests
                return result;
            }
            catch
            {
                // This one failed, wait for others
                if (tasks.Count == 0)
                    throw; // All failed
            }
        }

        throw new InvalidOperationException("All hedged requests failed");
    }

    private async Task<T> GetAsync<T>(HttpClient client, string path, CancellationToken ct)
    {
        var response = await client.GetAsync(path, ct);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadFromJsonAsync<T>(cancellationToken: ct)
            ?? throw new InvalidOperationException();
    }

    private Task<ProductDto?> GetFromCacheAsync(string? key) => Task.FromResult<ProductDto?>(null);
}

4. Health Checks & Diagnostics

4.1 Health Check System

[CODE] Basic Health Checks Setup:

var builder = WebApplication.CreateBuilder(args);

// Add health checks
builder.Services.AddHealthChecks()
    // Database check
    .AddSqlServer(
        connectionString: builder.Configuration.GetConnectionString("DefaultConnection")!,
        healthQuery: "SELECT 1",
        name: "sql-server",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "db", "sql", "ready" })

    // Redis check
    .AddRedis(
        redisConnectionString: builder.Configuration.GetConnectionString("Redis")!,
        name: "redis",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "cache", "ready" })

    // External API check
    .AddUrlGroup(
        uri: new Uri("https://api.external-service.com/health"),
        name: "external-api",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "external", "ready" })

    // Custom check
    .AddCheck<DiskSpaceHealthCheck>("disk-space", tags: new[] { "system" })
    .AddCheck<MemoryHealthCheck>("memory", tags: new[] { "system" });

var app = builder.Build();

// Map health endpoints
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    // Liveness: Is the app running?
    Predicate = _ => false, // No checks, just confirms app is running
    ResponseWriter = WriteMinimalResponse
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    // Readiness: Is the app ready to receive traffic?
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = WriteDetailedResponse
});

app.MapHealthChecks("/health/startup", new HealthCheckOptions
{
    // Startup: Has initialization completed?
    Predicate = check => check.Tags.Contains("startup"),
    ResponseWriter = WriteDetailedResponse
});

// Full health check endpoint (internal/admin only)
app.MapHealthChecks("/health", new HealthCheckOptions
{
    Predicate = _ => true,
    ResponseWriter = WriteDetailedResponse
}).RequireAuthorization("AdminOnly");

app.Run();

static Task WriteMinimalResponse(HttpContext context, HealthReport report)
{
    context.Response.ContentType = "text/plain";
    return context.Response.WriteAsync(report.Status.ToString());
}

static Task WriteDetailedResponse(HttpContext context, HealthReport report)
{
    context.Response.ContentType = "application/json";

    var response = new
    {
        status = report.Status.ToString(),
        totalDuration = report.TotalDuration.TotalMilliseconds,
        checks = report.Entries.Select(e => new
        {
            name = e.Key,
            status = e.Value.Status.ToString(),
            duration = e.Value.Duration.TotalMilliseconds,
            description = e.Value.Description,
            data = e.Value.Data,
            exception = e.Value.Exception?.Message
        })
    };

    return context.Response.WriteAsJsonAsync(response);
}

[CODE] Custom Health Checks:

public class DiskSpaceHealthCheck : IHealthCheck
{
    private readonly long _minimumFreeBytesThreshold;

    public DiskSpaceHealthCheck(long minimumFreeBytesThreshold = 1_073_741_824) // 1GB
    {
        _minimumFreeBytesThreshold = minimumFreeBytesThreshold;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken ct = default)
    {
        var drives = DriveInfo.GetDrives()
            .Where(d => d.IsReady && d.DriveType == DriveType.Fixed)
            .ToList();

        var unhealthyDrives = drives
            .Where(d => d.AvailableFreeSpace < _minimumFreeBytesThreshold)
            .ToList();

        var data = new Dictionary<string, object>();
        foreach (var drive in drives)
        {
            data[$"{drive.Name}_free_gb"] = drive.AvailableFreeSpace / 1_073_741_824.0;
            data[$"{drive.Name}_total_gb"] = drive.TotalSize / 1_073_741_824.0;
        }

        if (unhealthyDrives.Any())
        {
            return Task.FromResult(HealthCheckResult.Unhealthy(
                $"Low disk space on: {string.Join(", ", unhealthyDrives.Select(d => d.Name))}",
                data: data));
        }

        return Task.FromResult(HealthCheckResult.Healthy("Disk space OK", data));
    }
}

public class MemoryHealthCheck : IHealthCheck
{
    private readonly long _thresholdBytes;

    public MemoryHealthCheck(long thresholdBytes = 1_073_741_824) // 1GB
    {
        _thresholdBytes = thresholdBytes;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken ct = default)
    {
        var allocatedBytes = GC.GetTotalMemory(forceFullCollection: false);
        var memoryInfo = GC.GetGCMemoryInfo();

        var data = new Dictionary<string, object>
        {
            ["allocated_mb"] = allocatedBytes / 1_048_576.0,
            ["gen0_collections"] = GC.CollectionCount(0),
            ["gen1_collections"] = GC.CollectionCount(1),
            ["gen2_collections"] = GC.CollectionCount(2),
            ["heap_size_mb"] = memoryInfo.HeapSizeBytes / 1_048_576.0,
            ["fragmentation_percent"] = memoryInfo.FragmentedBytes * 100.0 / memoryInfo.HeapSizeBytes
        };

        if (allocatedBytes > _thresholdBytes)
        {
            return Task.FromResult(HealthCheckResult.Degraded(
                $"High memory usage: {allocatedBytes / 1_048_576}MB",
                data: data));
        }

        return Task.FromResult(HealthCheckResult.Healthy("Memory OK", data));
    }
}

public class DatabaseMigrationHealthCheck : IHealthCheck
{
    private readonly AppDbContext _context;

    public DatabaseMigrationHealthCheck(AppDbContext context)
    {
        _context = context;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken ct = default)
    {
        try
        {
            var pendingMigrations = await _context.Database
                .GetPendingMigrationsAsync(ct);

            var pending = pendingMigrations.ToList();

            if (pending.Any())
            {
                return HealthCheckResult.Unhealthy(
                    $"Pending migrations: {string.Join(", ", pending)}",
                    data: new Dictionary<string, object>
                    {
                        ["pending_count"] = pending.Count,
                        ["migrations"] = pending
                    });
            }

            return HealthCheckResult.Healthy("All migrations applied");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Cannot check migrations", ex);
        }
    }
}

// External service dependency check
public class ExternalApiHealthCheck : IHealthCheck
{
    private readonly HttpClient _httpClient;
    private readonly string _healthEndpoint;

    public ExternalApiHealthCheck(HttpClient httpClient, string healthEndpoint)
    {
        _httpClient = httpClient;
        _healthEndpoint = healthEndpoint;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken ct = default)
    {
        try
        {
            var sw = Stopwatch.StartNew();
            var response = await _httpClient.GetAsync(_healthEndpoint, ct);
            sw.Stop();

            var data = new Dictionary<string, object>
            {
                ["response_time_ms"] = sw.ElapsedMilliseconds,
                ["status_code"] = (int)response.StatusCode
            };

            if (response.IsSuccessStatusCode)
            {
                return sw.ElapsedMilliseconds > 5000
                    ? HealthCheckResult.Degraded("Slow response", data: data)
                    : HealthCheckResult.Healthy("OK", data);
            }

            return HealthCheckResult.Unhealthy(
                $"Unhealthy: {response.StatusCode}",
                data: data);
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Cannot reach service", ex);
        }
    }
}

4.2 Kubernetes Probes Integration

[PRODUCTION] K8s-Compatible Health Checks:

# kubernetes deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 80

        # Startup probe - called during initialization
        startupProbe:
          httpGet:
            path: /health/startup
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 30  # Allow 2.5 minutes for startup

        # Liveness probe - restart if fails
        livenessProbe:
          httpGet:
            path: /health/live
            port: 80
          initialDelaySeconds: 0
          periodSeconds: 10
          failureThreshold: 3

        # Readiness probe - remove from service if fails
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 80
          initialDelaySeconds: 0
          periodSeconds: 5
          failureThreshold: 3

4.3 Observability with OpenTelemetry

[CODE] OpenTelemetry Setup:

var builder = WebApplication.CreateBuilder(args);

// Add OpenTelemetry
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "MyService",
            serviceVersion: "1.0.0",
            serviceInstanceId: Environment.MachineName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
            options.Filter = context =>
                !context.Request.Path.StartsWithSegments("/health");
        })
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddSource("MyApp.CustomTracing")
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("http://otel-collector:4317");
        }))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddProcessInstrumentation()
        .AddMeter("MyApp.CustomMetrics")
        .AddOtlpExporter());

// Custom tracing
public class OrderService
{
    private static readonly ActivitySource Source = new("MyApp.CustomTracing");
    private static readonly Counter<long> OrderCounter =
        new Meter("MyApp.CustomMetrics").CreateCounter<long>("orders_created");

    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        using var activity = Source.StartActivity("CreateOrder");
        activity?.SetTag("customer_id", request.CustomerId);
        activity?.SetTag("items_count", request.Items.Count);

        try
        {
            var order = await ProcessOrderAsync(request);

            activity?.SetTag("order_id", order.Id);
            activity?.SetStatus(ActivityStatusCode.Ok);

            OrderCounter.Add(1, new KeyValuePair<string, object?>("status", "success"));

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);

            OrderCounter.Add(1, new KeyValuePair<string, object?>("status", "error"));

            throw;
        }
    }

    private Task<Order> ProcessOrderAsync(CreateOrderRequest request) =>
        Task.FromResult(new Order { Id = 1 });
}

public class CreateOrderRequest
{
    public int CustomerId { get; set; }
    public List<OrderItem> Items { get; set; } = new();
}

public class OrderItem { }
public class Order { public int Id { get; set; } }

5. Distributed Caching

5.1 IDistributedCache Abstraction

[CODE] Basic Distributed Caching:

var builder = WebApplication.CreateBuilder(args);

// Option 1: In-memory (development/single instance)
builder.Services.AddDistributedMemoryCache();

// Option 2: Redis (production)
builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = builder.Configuration.GetConnectionString("Redis");
    options.InstanceName = "MyApp:";
});

// Option 3: SQL Server
builder.Services.AddDistributedSqlServerCache(options =>
{
    options.ConnectionString = builder.Configuration.GetConnectionString("Cache");
    options.SchemaName = "dbo";
    options.TableName = "DistributedCache";
});

// Generic cache wrapper service
public interface ICacheService
{
    Task<T?> GetAsync<T>(string key, CancellationToken ct = default);
    Task SetAsync<T>(string key, T value, TimeSpan? expiration = null, CancellationToken ct = default);
    Task RemoveAsync(string key, CancellationToken ct = default);
    Task<T> GetOrSetAsync<T>(string key, Func<Task<T>> factory, TimeSpan? expiration = null, CancellationToken ct = default);
}

public class DistributedCacheService : ICacheService
{
    private readonly IDistributedCache _cache;
    private readonly ILogger<DistributedCacheService> _logger;
    private static readonly JsonSerializerOptions JsonOptions = new()
    {
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase
    };

    public DistributedCacheService(
        IDistributedCache cache,
        ILogger<DistributedCacheService> logger)
    {
        _cache = cache;
        _logger = logger;
    }

    public async Task<T?> GetAsync<T>(string key, CancellationToken ct = default)
    {
        try
        {
            var data = await _cache.GetStringAsync(key, ct);

            if (data == null)
            {
                _logger.LogDebug("Cache miss: {Key}", key);
                return default;
            }

            _logger.LogDebug("Cache hit: {Key}", key);
            return JsonSerializer.Deserialize<T>(data, JsonOptions);
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "Cache get error for {Key}", key);
            return default;
        }
    }

    public async Task SetAsync<T>(
        string key,
        T value,
        TimeSpan? expiration = null,
        CancellationToken ct = default)
    {
        try
        {
            var options = new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = expiration ?? TimeSpan.FromMinutes(5),
                SlidingExpiration = TimeSpan.FromMinutes(1)
            };

            var data = JsonSerializer.Serialize(value, JsonOptions);
            await _cache.SetStringAsync(key, data, options, ct);

            _logger.LogDebug("Cache set: {Key}", key);
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "Cache set error for {Key}", key);
        }
    }

    public async Task RemoveAsync(string key, CancellationToken ct = default)
    {
        try
        {
            await _cache.RemoveAsync(key, ct);
            _logger.LogDebug("Cache removed: {Key}", key);
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "Cache remove error for {Key}", key);
        }
    }

    public async Task<T> GetOrSetAsync<T>(
        string key,
        Func<Task<T>> factory,
        TimeSpan? expiration = null,
        CancellationToken ct = default)
    {
        var cached = await GetAsync<T>(key, ct);
        if (cached != null)
            return cached;

        var value = await factory();
        await SetAsync(key, value, expiration, ct);
        return value;
    }
}

5.2 Cache Stampede Prevention

[CODE] Preventing Cache Stampede:

public class StampedePreventingCacheService : ICacheService
{
    private readonly IDistributedCache _cache;
    private readonly ConcurrentDictionary<string, SemaphoreSlim> _locks = new();
    private readonly ILogger<StampedePreventingCacheService> _logger;

    public StampedePreventingCacheService(
        IDistributedCache cache,
        ILogger<StampedePreventingCacheService> logger)
    {
        _cache = cache;
        _logger = logger;
    }

    public async Task<T> GetOrSetAsync<T>(
        string key,
        Func<Task<T>> factory,
        TimeSpan? expiration = null,
        CancellationToken ct = default)
    {
        // Try cache first (no lock)
        var cached = await GetAsync<T>(key, ct);
        if (cached != null)
            return cached;

        // Get or create lock for this key
        var keyLock = _locks.GetOrAdd(key, _ => new SemaphoreSlim(1, 1));

        // Only one thread will execute factory
        await keyLock.WaitAsync(ct);
        try
        {
            // Double-check after acquiring lock
            cached = await GetAsync<T>(key, ct);
            if (cached != null)
                return cached;

            _logger.LogDebug("Cache miss, executing factory for {Key}", key);

            var value = await factory();
            await SetAsync(key, value, expiration, ct);
            return value;
        }
        finally
        {
            keyLock.Release();

            // Clean up lock if not being waited on
            if (keyLock.CurrentCount == 1)
            {
                _locks.TryRemove(key, out _);
            }
        }
    }

    // Implement other methods...
    public Task<T?> GetAsync<T>(string key, CancellationToken ct = default) =>
        Task.FromResult<T?>(default);

    public Task SetAsync<T>(string key, T value, TimeSpan? expiration = null, CancellationToken ct = default) =>
        Task.CompletedTask;

    public Task RemoveAsync(string key, CancellationToken ct = default) =>
        Task.CompletedTask;
}

// Alternative: Early refresh (probabilistic)
public class EarlyRefreshCacheService
{
    private readonly IDistributedCache _cache;
    private readonly double _earlyRefreshProbability;

    public EarlyRefreshCacheService(IDistributedCache cache, double earlyRefreshProbability = 0.1)
    {
        _cache = cache;
        _earlyRefreshProbability = earlyRefreshProbability;
    }

    public async Task<T?> GetWithEarlyRefreshAsync<T>(
        string key,
        Func<Task<T>> factory,
        TimeSpan expiration,
        CancellationToken ct = default)
    {
        var dataKey = $"data:{key}";
        var expiryKey = $"expiry:{key}";

        var data = await _cache.GetStringAsync(dataKey, ct);
        var expiryStr = await _cache.GetStringAsync(expiryKey, ct);

        if (data != null && expiryStr != null)
        {
            var expiry = DateTimeOffset.Parse(expiryStr);
            var remaining = expiry - DateTimeOffset.UtcNow;
            var original = expiration;

            // Probabilistic early refresh
            // As we get closer to expiry, probability of refresh increases
            var probabilityOfRefresh = 1 - (remaining.TotalSeconds / original.TotalSeconds);

            if (Random.Shared.NextDouble() < probabilityOfRefresh * _earlyRefreshProbability)
            {
                // Trigger background refresh
                _ = Task.Run(async () =>
                {
                    var value = await factory();
                    await SetWithExpiryAsync(dataKey, expiryKey, value, expiration, ct);
                }, ct);
            }

            return JsonSerializer.Deserialize<T>(data);
        }

        // Cache miss - fetch and store
        var newValue = await factory();
        await SetWithExpiryAsync(dataKey, expiryKey, newValue, expiration, ct);
        return newValue;
    }

    private async Task SetWithExpiryAsync<T>(
        string dataKey,
        string expiryKey,
        T value,
        TimeSpan expiration,
        CancellationToken ct)
    {
        var options = new DistributedCacheEntryOptions
        {
            AbsoluteExpirationRelativeToNow = expiration
        };

        var data = JsonSerializer.Serialize(value);
        var expiry = DateTimeOffset.UtcNow.Add(expiration).ToString("O");

        await Task.WhenAll(
            _cache.SetStringAsync(dataKey, data, options, ct),
            _cache.SetStringAsync(expiryKey, expiry, options, ct));
    }
}

5.3 Multi-Tier Caching

[CODE] L1 (Memory) + L2 (Redis) Caching:

public class MultiTierCacheService : ICacheService
{
    private readonly IMemoryCache _l1Cache;
    private readonly IDistributedCache _l2Cache;
    private readonly ILogger<MultiTierCacheService> _logger;

    private readonly MemoryCacheEntryOptions _l1Options = new()
    {
        AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(1),
        Size = 1
    };

    public MultiTierCacheService(
        IMemoryCache l1Cache,
        IDistributedCache l2Cache,
        ILogger<MultiTierCacheService> logger)
    {
        _l1Cache = l1Cache;
        _l2Cache = l2Cache;
        _logger = logger;
    }

    public async Task<T?> GetAsync<T>(string key, CancellationToken ct = default)
    {
        // L1: In-process memory (fastest)
        if (_l1Cache.TryGetValue(key, out T? l1Value))
        {
            _logger.LogDebug("L1 cache hit: {Key}", key);
            return l1Value;
        }

        // L2: Distributed cache
        try
        {
            var l2Data = await _l2Cache.GetStringAsync(key, ct);
            if (l2Data != null)
            {
                _logger.LogDebug("L2 cache hit: {Key}", key);

                var value = JsonSerializer.Deserialize<T>(l2Data);

                // Populate L1 for next request
                _l1Cache.Set(key, value, _l1Options);

                return value;
            }
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "L2 cache error for {Key}", key);
        }

        _logger.LogDebug("Cache miss: {Key}", key);
        return default;
    }

    public async Task SetAsync<T>(
        string key,
        T value,
        TimeSpan? expiration = null,
        CancellationToken ct = default)
    {
        var exp = expiration ?? TimeSpan.FromMinutes(5);

        // Set L1
        _l1Cache.Set(key, value, new MemoryCacheEntryOptions
        {
            AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(1), // Shorter for L1
            Size = 1
        });

        // Set L2
        try
        {
            var data = JsonSerializer.Serialize(value);
            await _l2Cache.SetStringAsync(key, data, new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = exp
            }, ct);
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "L2 cache set error for {Key}", key);
        }
    }

    public async Task RemoveAsync(string key, CancellationToken ct = default)
    {
        _l1Cache.Remove(key);

        try
        {
            await _l2Cache.RemoveAsync(key, ct);
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "L2 cache remove error for {Key}", key);
        }
    }

    public async Task<T> GetOrSetAsync<T>(
        string key,
        Func<Task<T>> factory,
        TimeSpan? expiration = null,
        CancellationToken ct = default)
    {
        var cached = await GetAsync<T>(key, ct);
        if (cached != null)
            return cached;

        var value = await factory();
        await SetAsync(key, value, expiration, ct);
        return value;
    }
}

// DI Registration
public static class CacheServiceExtensions
{
    public static IServiceCollection AddMultiTierCaching(
        this IServiceCollection services,
        IConfiguration config)
    {
        // L1: Memory cache with size limit
        services.AddMemoryCache(options =>
        {
            options.SizeLimit = 10000; // Max entries
        });

        // L2: Redis
        services.AddStackExchangeRedisCache(options =>
        {
            options.Configuration = config.GetConnectionString("Redis");
            options.InstanceName = "MyApp:";
        });

        services.AddSingleton<ICacheService, MultiTierCacheService>();

        return services;
    }
}

6. Interview Questions & Answers

CQRS

Q: When should you NOT use CQRS? A: CQRS adds complexity and should be avoided when:

Simple CRUD applications with low complexity
No scaling requirements between reads and writes
Team unfamiliar with the pattern
Small applications where the overhead isn’t justified
Real-time consistency is critical (eventual consistency is unacceptable)

Q: How do you keep read and write models in sync? A: Several approaches:

Synchronous: Update both in the same transaction (simplest, tightly coupled)
Domain Events: Publish events from write side, handlers update read models
Event Sourcing: Read models rebuilt from event stream
Change Data Capture (CDC): Database-level change tracking (Debezium, etc.)

Rate Limiting

Q: Which rate limiting algorithm would you choose for an API? A: It depends on the use case:

Fixed Window: Simple, predictable, but allows boundary bursts
Sliding Window: Smooth limiting, no bursts, higher memory usage
Token Bucket: Allow controlled bursts, good for APIs with variable traffic
Concurrency: When you need to limit parallel execution, not rate

Q: How do you implement rate limiting in a distributed system? A: Use a centralized store (Redis) with atomic operations:

Lua scripts for atomicity
Sorted sets for sliding window
Consider using dedicated libraries (Redis Rate Limiter)
Handle cache failures gracefully (fail open vs fail closed)

Resilience

Q: What’s the difference between retry and circuit breaker? A:

Retry: Repeats failed operations, useful for transient failures
Circuit Breaker: Prevents calls to failing services, allows recovery time
Use together: Circuit breaker wraps retry to prevent retry storms

Q: How do you choose timeout values? A:

Per-try timeout: Slightly longer than P99 latency (e.g., 5-10 seconds)
Total timeout: Per-try × retries + buffer (e.g., 30-60 seconds)
Consider downstream timeouts (don’t exceed them)
Use hedging for latency-sensitive operations

Health Checks

Q: What’s the difference between liveness and readiness probes? A:

Liveness: Is the process running? Failure = restart container
Readiness: Can the app handle traffic? Failure = remove from load balancer
Startup: Has initialization completed? Prevents premature liveness checks

Caching

Q: What is cache stampede and how do you prevent it? A: Cache stampede occurs when many requests simultaneously try to regenerate an expired cache entry.

Prevention strategies:

Locking: Only one request regenerates, others wait
Probabilistic early refresh: Randomly refresh before expiry
Background refresh: Separate process refreshes cache
Stale-while-revalidate: Serve stale data while refreshing

Q: How do you invalidate cache across multiple instances? A:

Pub/Sub: Redis pub/sub or message queue to notify all instances
Short TTL: Accept some staleness, rely on expiration
Versioned keys: Change key when data changes
Cache tags: Group related entries, invalidate by tag

Summary

This guide covered essential modern .NET patterns for building robust, scalable applications:

Pattern	Key Benefit	When to Use
CQRS	Separate optimization of reads/writes	High read/write ratio, complex queries
Rate Limiting	Protect resources from overload	Public APIs, shared resources
Resilience (Polly)	Handle transient failures gracefully	External dependencies, microservices
Health Checks	Monitor application health	Kubernetes deployments, load balancers
Distributed Caching	Reduce latency and database load	Frequently accessed data, scaling

Best Practices Checklist:

[ ] Start simple, add complexity only when needed
[ ] Measure before optimizing
[ ] Implement proper monitoring and alerting
[ ] Test failure scenarios (chaos engineering)
[ ] Document architectural decisions
[ ] Consider operational complexity alongside technical benefits

Modern Dotnet Patterns Guide