43 lines
2.1 KiB
Go
43 lines
2.1 KiB
Go
|
// Copyright (c) 2026 Petr Balvín <opensource@petrbalvin.org> (https://petrbalvin.org)
|
||
|
|
// SPDX-License-Identifier: MIT
|
||
|
|
|
||
|
|
// Package spmd runs one program on many ranks: explicit SPMD worlds
|
||
|
|
// with the collectives of scientific distributed computing, over TCP
|
||
|
|
// between machines or in process within one.
|
||
|
|
//
|
||
|
|
// # The determinism contract
|
||
|
|
//
|
||
|
|
// Every collective answers bit-identical results whatever the world
|
||
|
|
// size, the machine, the worker count or the order messages arrive in.
|
||
|
|
// The rule that buys this is one: the order of a reduction is a
|
||
|
|
// function of the data, never of the topology. Family B reductions
|
||
|
|
// (ReduceShards, AllReduceShards) cut the global length at the fold's
|
||
|
|
// own block boundaries, fold each block where its elements live, and
|
||
|
|
// combine the block partials with the same balanced tree the
|
||
|
|
// single-array fold uses, so a sharded reduction and the single-array
|
||
|
|
// Sum are one computation and one answer, for one rank or for fifty.
|
||
|
|
// Family A reductions (Reduce, AllReduce) fold same-shaped arrays in
|
||
|
|
// rank index order, which the program itself fixes. No collective
|
||
|
|
// every combines partials in the order they happen to arrive.
|
||
|
|
//
|
||
|
|
// # Worlds
|
||
|
|
//
|
||
|
|
// Launch builds a world of size ranks in one process, one goroutine
|
||
|
|
// per rank, over channels: the development surface, the test surface
|
||
|
|
// and the single-machine surface. Listen and Join build the same world
|
||
|
|
// over TCP, rank 0 listening and every other rank dialling; ranks are
|
||
|
|
// assigned in dial order. The sharded reductions' results never
|
||
|
|
// depend on the assignment; the same-shaped arrays' fold order is the
|
||
|
|
// rank order the program fixes, so a program that wants bit-reproducible
|
||
|
|
// family A runs keeps its rank assignment stable. Collectives are bulk synchronous: every rank calls the
|
||
|
|
// same collective in the same order, as an MPI program does.
|
||
|
|
//
|
||
|
|
// Any failure, a deadline included, fails the whole world: the rank
|
||
|
|
// that saw it and every rank that then touches the world or waits on
|
||
|
|
// it receive an error, and no collective ever returns a partial
|
||
|
|
// numeric result. A failed world stays failed.
|
||
|
|
//
|
||
|
|
// The package is pure Go on the standard library: no cgo, no
|
||
|
|
// dependency, no build tag.
|
||
|
|
package spmd
|