Source content from a remote API

To build pages from a remote HTTP API instead of local files — a release feed, a CMS, a product catalog behind a JSON endpoint — implement IContentService over a typed HttpClient. This guide is the remote counterpart to Source content from outside the markdown pipeline: that page covers the discovery, TOC, and cross-reference work every content service shares. This one adds the four things a network source needs:

• awaiting HTTP in DiscoverAsync,
• caching one fetch across every pipeline pass,
• rendering markdown bodies that arrive over the wire,
• and surviving a slow or unreachable API at build time.

The recipe references examples/BeyondRemoteContentExample, which turns the GitHub Releases API into /releases/{version}/ pages.

Before you begin

• A working Pennington site on bare AddPennington (see Create your first Pennington site).
• The discovery shape from Source content from outside the markdown pipeline — this guide assumes you know how DiscoverAsync, GetContentTocEntriesAsync, and GetCrossReferencesAsync fit together, and focuses on what changes when the source is remote.
• An API reachable over HTTP that returns JSON. The example uses an unauthenticated public endpoint; for an authenticated API, add the token to the typed client's default headers.

Fetch the data with a typed HttpClient

Register a typed HttpClient with AddHttpClient<T>. Set a User-Agent — the GitHub API answers 403 without one — and a Timeout, so a stalled API cannot hang the build:

builder.Services.AddHttpClient<GitHubReleasesClient>(client =>
{
    client.BaseAddress = new Uri("https://api.github.com/");
    client.DefaultRequestHeaders.UserAgent.ParseAdd("Pennington-Remote-Content-Example");
    client.Timeout = TimeSpan.FromSeconds(10);
});

The client itself is a thin wrapper that fetches and deserializes. GetFromJsonAsync with a snake-case naming policy maps GitHub's tag_name/html_url/published_at onto a PascalCase record:

public async Task<ImmutableList<GitHubRelease>> GetReleasesAsync()
{
    try
    {
        // The User-Agent header (set in Program.cs) is required — GitHub answers
        // 403 without one. per_page caps the page; a busy repo paginates via the
        // response Link header.
        var releases = await _http.GetFromJsonAsync<List<GitHubRelease>>(
            $"repos/{Owner}/{Repo}/releases?per_page=20", JsonOptions);
  
        if (releases is { Count: > 0 })
        {
            return [.. releases.Where(r => !r.Draft)];
        }
  
        _logger.LogWarning("GitHub returned no releases; using the bundled fixture.");
    }
    catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException or JsonException)
    {
        // Fail open: a slow, unreachable, or rate-limited API must not break the
        // build. To fail the build instead, rethrow here and let it propagate.
        _logger.LogWarning(ex, "GitHub releases fetch failed; using the bundled fixture.");
    }
  
    return await LoadFixtureAsync();
}

The try/catch is the build-time failure boundary — covered under Handle a slow or unreachable API below.

Cache the fetch across every pass

DiscoverAsync, the TOC pass, the cross-reference pass, and the rendering endpoint each need the data. Fetching in each one would hit the API four-plus times per build. Cache the result in an AsyncLazy<T> (from Pennington.Infrastructure) created once in the constructor, and await it everywhere:

private readonly AsyncLazy<ImmutableList<ReleaseEntry>> _entriesLazy;
  
public GitHubReleasesContentService(GitHubReleasesClient client)
    => _entriesLazy = new AsyncLazy<ImmutableList<ReleaseEntry>>(() => LoadAsync(client));

AsyncLazy<T> runs its factory once on first access and replays the same task to every later caller; a faulted fetch is evicted so the next access retries. DiscoverAsync awaits it like every other pass, pairing each route with EndpointSource so the build crawler fetches the URL through the endpoint below:

public async IAsyncEnumerable<DiscoveredItem> DiscoverAsync()
{
    yield return new DiscoveredItem(
        ContentRouteFactory.FromUrl(new UrlPath("/releases/")),
        new EndpointSource());
  
    foreach (var entry in await _entriesLazy)
    {
        yield return new DiscoveredItem(
            ContentRouteFactory.FromUrl(new UrlPath($"/releases/{entry.Version}/")),
            new EndpointSource());
    }
}

Because this service reads no local files, there is nothing to file-watch — register it as a process-lifetime singleton (see Register the service). A service backed by files that change during a dev session needs the file-watched lifetimes described in Publish a custom feed from a content service, or it serves stale data.

Render the API's markdown body yourself

GitHub returns each release's notes as markdown. EndpointSource routes are not run through Markdig automatically — the framework hands the route to your endpoint and renders nothing — so call IContentRenderer yourself. Build a ParsedItem from the markdown and render it; the result's Content.Html is the same output a markdown file would produce:

public static async Task<string> RenderDetailAsync(IContentRenderer renderer, ReleaseEntry entry)
{
    var route = ContentRouteFactory.FromUrl(new UrlPath($"/releases/{entry.Version}/"));
    var parsed = new ParsedItem(route, new ReleaseFrontMatter(entry.Title), entry.BodyMarkdown ?? "");
    var rendered = await renderer.RenderAsync(parsed);
  
    var body = rendered.Value is RenderedItem item
        ? item.Content.Html
        : "<p>Release notes are unavailable.</p>";
  
    return Page(entry.Title, $"""
        <p class="meta">
          <time datetime="{entry.Date:yyyy-MM-dd}">{entry.Date:yyyy-MM-dd}</time>
          &middot; <a href="{entry.HtmlUrl}">View on GitHub</a>
        </p>
        {body}
        """);
}

Wrap that HTML in the element named by SiteProjection.ContentSelector. Set the selector to match that element in the AddPennington callback — here both are <article>:

builder.Services.AddPennington(penn =>
{
    // The endpoint below wraps each release body in <article>; point the
    // projection selector at that element.
    penn.SiteProjection.ContentSelector = "article";
});

At build time the projection self-fetches every TOC-listed route through the live pipeline and splits the selected element into heading sections — so an EndpointSource page rendered this way is indexed at heading level for search and llms.txt, exactly like a markdown page. Without the selector match, the page chrome leaks into the index instead of the release body.

Register the service

Register the concrete type, then forward IContentService to the same instance so the endpoint and the pipeline share one cache:

builder.Services.AddSingleton<GitHubReleasesContentService>();
builder.Services.AddSingleton<IContentService>(sp =>
    sp.GetRequiredService<GitHubReleasesContentService>());

A singleton holding a typed HttpClient is normally discouraged — the pooled handler stops rotating, so long-lived processes can pin stale DNS. Here it is fine: the client is used for a single fetch at startup and never again; the cache, not the client, is what lives for the process. For a long-running host that re-fetches periodically, inject IHttpClientFactory and create a client per fetch instead.

Handle a slow or unreachable API at build time

A build that fetches from the network inherits the network's failure modes: the API can be down, rate-limited, or slow. The Timeout on the typed client bounds the slow case. For the rest, GetReleasesAsync fails open — on HttpRequestException, a timeout, or malformed JSON it logs a warning and falls back to a committed fixture (fixtures/github-releases.json), so an offline or rate-limited build still produces a complete site. The fixture also makes the build deterministic in CI.

To fail closed instead — stop the build when the data is unavailable rather than ship a snapshot that may be stale or empty — rethrow from the catch and let it propagate out of DiscoverAsync. Choose per site: a marketing build might prefer last-known-good data; a compliance build might prefer to fail loudly.

Keep the published data fresh

For content that updates on its own schedule, rebuild on a schedule. The only thing a scheduled rebuild adds to an ordinary deploy workflow is a schedule trigger — a cron entry that fires the same build-and-deploy job on a timer instead of (or alongside) a push:

on:
  schedule:
    - cron: "0 6 * * *"   # rebuild nightly at 06:00 UTC
  workflow_dispatch:       # plus a manual trigger

The job that this trigger runs — checkout, setup-dotnet, dotnet run -- build, and the deploy steps — is the same one a push build uses. Build it from Deploy to GitHub Pages and add the schedule trigger above.

Verify

• Run dotnet run --project examples/BeyondRemoteContentExample and open /releases/. The index lists every release and each /releases/{version}/ renders its notes as formatted HTML (headings, lists, links), not raw markdown.
• Run dotnet run --project examples/BeyondRemoteContentExample -- build output. Confirm output/releases/ has one folder per release, the rendered markdown carries <h2> headings, output/search/ indexes those heading texts (proving the EndpointSource bodies reach search via the self-fetch), and output/sitemap.xml lists every /releases/{version}/ URL.
• Disconnect from the network and rebuild. The build still succeeds, serving the fixture snapshot.

Related

• How-to: Source content from outside the file system — the base IContentService shape this guide builds on.
• How-to: Publish a custom feed from a content service — the DI lifetimes for a file-backed service, and the stale-cache trap.
• How-to: Publish a sitemap — what the sitemap includes and excludes.
• Background: Why ContentSource is a union — what EndpointSource means and why its pages are still listed in the sitemap.