Semantic Caching for ASP.NET Core AI APIs: Cut Latency and Model Costs Safely

AI-powered endpoints often have two expensive characteristics: they call a remote model and they receive the same question more than once. A semantic cache helps with both. Instead of looking only for an identical string, it reuses a prior answer when a new question means nearly the same thing.

This tutorial shows a practical pattern for adding semantic caching to an ASP.NET Core API without putting cache rules inside the endpoint. The example uses interfaces so you can connect your preferred embeddings and cache provider later.

When semantic caching is a good fit

Use it for read-like AI experiences: documentation assistants, product Q&A, support suggestions, and internal knowledge search. Avoid it for requests where freshness, user-specific permissions, or exact wording materially changes the answer.

A cache hit should be an optimization, not an authorization shortcut. Include tenant, user scope, language, prompt version, and relevant data version in the cache key or filter.

Model the cache boundary

Keep the application service responsible for the workflow and hide provider details behind small interfaces. The cache returns a hit only when its similarity score is above your chosen threshold.

public sealed record CachedAnswer(
    string Text,
    double Similarity);

public interface ISemanticAnswerCache
{
    Task<CachedAnswer?> FindAsync(
        string scope,
        string question,
        CancellationToken cancellationToken);

    Task StoreAsync(
        string scope,
        string question,
        string answer,
        CancellationToken cancellationToken);
}

public interface IAiAnswerClient
{
    Task<string> AnswerAsync(
        string question,
        CancellationToken cancellationToken);
}

A real implementation can create an embedding for the question, search a vector-capable store, and return both the answer and cosine similarity. This interface keeps that infrastructure choice out of your API layer.

Implement cache-first orchestration

The service checks the cache before making an AI call. Notice that it requires a scope: answers generated for one tenant should not become visible to another tenant.

public sealed class AnswerService
{
    private const double MinimumSimilarity = 0.92;
    private readonly ISemanticAnswerCache _cache;
    private readonly IAiAnswerClient _aiClient;

    public AnswerService(
        ISemanticAnswerCache cache,
        IAiAnswerClient aiClient)
    {
        _cache = cache;
        _aiClient = aiClient;
    }

    public async Task<AnswerResult> GetAnswerAsync(
        string tenantId,
        string question,
        CancellationToken cancellationToken)
    {
        var scope = $"{tenantId}:help-v3:en";
        var cached = await _cache.FindAsync(scope, question, cancellationToken);

        if (cached is { Similarity: >= MinimumSimilarity })
        {
            return new AnswerResult(cached.Text, FromCache: true);
        }

        var answer = await _aiClient.AnswerAsync(question, cancellationToken);
        await _cache.StoreAsync(scope, question, answer, cancellationToken);

        return new AnswerResult(answer, FromCache: false);
    }
}

public sealed record AnswerResult(string Text, bool FromCache);

Expose a thin ASP.NET Core endpoint

The endpoint validates the request, derives the scope from the authenticated user, and lets the service decide whether the response is cached. Do not accept a tenant identifier directly from an untrusted request body.

app.MapPost("/api/assistant/answer", async (
    AskRequest request,
    ClaimsPrincipal user,
    AnswerService answers,
    CancellationToken cancellationToken) =>
{
    if (string.IsNullOrWhiteSpace(request.Question) || request.Question.Length > 2_000)
    {
        return Results.BadRequest(new { error = "Question must be 1 to 2000 characters." });
    }

    var tenantId = user.FindFirst("tenant_id")?.Value;
    if (string.IsNullOrWhiteSpace(tenantId))
    {
        return Results.Forbid();
    }

    var result = await answers.GetAnswerAsync(
        tenantId,
        request.Question,
        cancellationToken);

    return Results.Ok(result);
}).RequireAuthorization();

public sealed record AskRequest(string Question);

Choose the similarity threshold with evidence

There is no universal correct number. Start conservatively, such as 0.92, and collect a review set of question pairs from your real domain. Raise the threshold if mismatched answers appear; lower it only after confirming you are missing safe reuse opportunities.

Store the prompt template version and the knowledge-base version alongside each cache entry. When either changes, invalidate old entries or change the scope prefix. This prevents an answer produced with an old policy or document set from silently surviving a release.

Protect correctness, privacy, and cost

• Separate scopes: Never search across tenants or permission boundaries.
• Set an expiry: Use a TTL appropriate to how often the underlying information changes.
• Limit stored data: Avoid saving secrets, personal data, or raw confidential prompts unless you have a documented reason and retention policy.
• Measure hit quality: Track cache-hit rate, similarity distribution, latency, token savings, and user corrections.
• Use a fallback: If vector search is unavailable, call the AI provider instead of returning an uncertain answer.

Production checklist

• Version the prompt and knowledge source in the cache scope.
• Cap answer size and cache-entry TTL.
• Log cache decisions without logging sensitive question text.
• Test isolation between tenants and roles.
• Monitor cache misses after prompt or content deployments.
• Keep rate limits around the endpoint; a cache reduces cost but does not stop abuse.

Final thoughts

Semantic caching is most valuable when it is treated as a reliability and cost-control layer, not a shortcut around access control. With strict scopes, versioned entries, measured thresholds, and a clean ASP.NET Core boundary, you can deliver faster AI responses while reducing repeat model calls safely.

Related CodeWithChai resources

• Build an AI chatbot API with clean architecture.
• Configure and structure a .NET 10 API with dependency injection.
• Add production health checks to an ASP.NET Core API.