Contributing

The rules that matter

Two packages are pure: packages/parsers and packages/calendar. They take strings and return typed records. No fetch, no filesystem, no Hono, no Cloudflare types, no process.env, no wall clock.

bun run purity

This is a build failure, not a review comment. Without it the rule erodes in month three, when a source needs "just one" redirect follow and adding a fetch looks easier than moving the call up into the pipeline. Then the parser can only be tested against a live upstream, the tests go flaky, someone marks them skipped, and the most valuable property of the codebase is gone.

Network I/O lives only in packages/pipeline and its callers.

Adding a source

1. Register it in packages/schema/src/sources.ts with its licence, attribution, retention class, freshness SLO, and whether it needs an Indian IP. The registry drives SOURCES.md, the raw-archive retention, and the health reporting in meta.json, nothing is configured twice.

2. Add fixtures in fixtures/{source_id}/. At minimum:

The last two matter most. A 200 with a stale payload is the failure that looks like success, and an HTML error page parsed leniently becomes a silent zero.

3. Write the parser in packages/parsers/src/. It returns a ParseResult:

export interface ParseResult {
  source_id: string;
  points: RawPoint[];
  latest_date: string | null;
  warnings: string[];
}

A parser deliberately does not decide is_trading_day, status, or filled_from. Those come from packages/calendar and the normaliser. Its only job is: what did this upstream body actually say?

Fail loudly. An unparseable body must throw, never return an empty result, "no data today" and "the endpoint moved" must not look the same.

Never coerce a placeholder to zero. parseNumber already rejects ., .., -, n/a and friends.

4. Add instruments in packages/schema/src/instruments.ts and a coverage entry in coverage.ts with an honest limit_reason. Endpoints are derived from the instrument registry, so adding an instrument adds its URLs, there is no route table to update.

5. Regenerate and commit:

bun run generate

The CI gates

bun run check-types
bun test
bun run purity            # parser purity
bun run generate          # then `git diff --exit-code`
bun run test:idempotency  # publish twice, assert byte-identical

Three of these are specific to this project and worth more than the standard three:

Parser purity turns the most important rule in the codebase from a convention into a build failure.

Schema/OpenAPI drift regenerates the JSON Schema, openapi.json and SOURCES.md from Zod, then diffs. This is what makes "one source of truth, several artifacts" true rather than aspirational: a hand-maintained spec drifts within a month.

Publish idempotency runs publish twice against a frozen fixture date and asserts byte-identical output. Idempotency is what makes three uncoordinated schedulers safe; if it silently breaks you get duplicate commits and cache thrash with no other symptom.

Data hygiene

Non-negotiable, and enforced where possible rather than documented:

• Never commit a value obtained from IBJA or any relay of it.
• Never redistribute full exchange bhavcopy files. Index levels only.
• Never store secrets in attempted_url or log credential headers, redactUrl exists for this and every audit write goes through it.
• Every published field maps to an entry in SOURCES.md, generated from schema.

Running it locally

bun install
bun run ingest --fixtures   # offline, frozen clock, reproducible
bun test

--fixtures runs the whole pipeline against fixtures/ with a frozen clock, so the output is byte-reproducible and therefore diffable in a pull request.