Mirror a paginated API to NDJSON
Pull every page of a cursor-paginated, OAuth2-protected endpoint — with retry and throttle on each page — and write the rows as newline-delimited JSON.
Task
You want a local copy of every record behind a paginated API — one that happens
to be OAuth2-protected, rate-limited, and prone to the odd 429 or 503. By
hand that means a token dance, a manual page loop, backoff code, and a rate
limiter before you write a single row.
Example
One way: declare all four concerns on a single stitch and await it once.
import { , , } from 'stitchapi';
const = ({
: 'https://api.example.com',
: '/users',
: ({
: 'https://api.example.com/oauth/token',
: ('CLIENT_ID'),
: ('CLIENT_SECRET'),
: 'users.read',
}),
: {
: 4,
: [429, 503],
: 'expo-jitter',
: true,
},
: { : '5/s', : 2, : 'host' },
: {
: () => {
const = ( as { ?: string }).;
return ? { : { } } : ;
},
: () => ( as { : unknown[] }).,
: 100,
},
});
// One await runs the whole paged loop — auth, retry, and throttle apply to
// every page — and aggregates every row into a single array.
const = (await ()) as unknown[];
// Newline-delimited JSON: one row per line.
const = .(() => .()).('\n');Write the rows to disk:
import { writeFile } from 'node:fs/promises';
await writeFile('users.ndjson', ndjson);How it works
The four concerns are independent keys on one declaration, so they compose
without any glue. oauth2 fetches, caches, and
refreshes the token behind the
capability boundary — your code
never sees it. paginate reads nextCursor
off each raw body and asks for the next page until next returns undefined,
aggregating every page's results into the array you await.
retry and
throttle apply per page, not once for
the whole run: a 429 on page seven recovers on its own, and the loop stays
under five requests a second the entire time. Without an
output schema the rows arrive as
unknown[], so the cast just tells TypeScript what you already know — add a
schema and each row is typed and validated, and the cast goes away. Either way,
serializing to NDJSON is one .map().
next reads the raw body; items reads the value after unwrap. Reach for
the cursor in next from where it actually lives in the response — see
Pagination.
See also
Watch retries and throttling as they happen
Iterate a stitch's event stream instead of awaiting it, and observe every retry, pause, and drift in real time.
Share one rate limit across every worker
Point the throttle at a shared store so a whole fleet draws from a single rate budget instead of N× the limit.