API reference

The portable public surface lives in five headers:

Two further headers are backend-specific and compile only in a build that enables that backend:

All preconditions and dispatch failures throw std::runtime_error with a "brotensor: <op>: <reason>" message.

Tensor

A row-major (rows, cols) buffer with runtime Dtype + Device tags. Copyable (device-aware deep copy) and movable. Rank-1 data is (N, 1); higher-rank layouts are flattened per each op's shape contract.

enum class Device { CPU, CUDA, Metal };
enum class Dtype  { FP32, FP16, INT8, INT32, BF16, F64,
                    Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1,        // GGUF legacy blocks
                    Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q8_K };       // GGUF K-quant superblocks

Factories

Migration and readback

resize() reuses storage whenever the requested shape fits the existing allocation — capacity is the high-water mark of the tensor's past sizes — and reallocates only when growing past it. A no-op when the shape and dtype already match. So a scratch buffer cycling through shapes stabilises at its largest size instead of reallocating every call, and its device pointer stays stable — which is what makes a tensor reusable across a CUDA-graph-captured op sequence.

Call sync(device) / sync_all() before reading GPU results back to the host — GPU ops are asynchronous.

CPU-resident tensors additionally expose direct host accessors (host_f32_mut(), at(), operator[], …) — see tensor.h.

Dtype helpers

Free functions for sizing a buffer without special-casing the quant carriers:

Bit-conversion helpers

fp32_to_fp16_bits / fp16_bits_to_fp32 / fp32_to_bf16_bits / bf16_bits_to_fp32 — pure-CPU scalar conversions between FP32 and half/bfloat bit patterns, for tests and small host-side preprocessing.

Runtime

Shutdown. The CPU backend's worker threads otherwise live until the thread pool's Meyers-singleton destructor runs during static destruction, by which point the OS has already suspended every other thread. A worker suspended mid-op while holding a global lock (e.g. the Debug CRT's iterator-checking mutex) can deadlock the main thread's own exit-time destructors on that same lock. shutdown() makes the teardown deterministic instead.

Trimming. device_mem_trim is worth calling between pipeline phases with very different scratch shapes: cached blocks count against device residency, and on Windows (WDDM) sustained near-full commit makes the OS demote large resident allocations to shared memory — silently turning weight reads into PCIe traffic.

safetensors (<brotensor/safetensors.h>)

mmap'd zero-copy reader plus a writer. Namespace brotensor::safetensors.

• File — opens and mmaps a .safetensors file, parses the JSON header, exposes tensors by name as TensorViews (name, dtype, shape, raw byte span).
• Upload helpers (view → device Tensor). All require an F32 / F16 / BF16 source view; brotensor is 2D-only, so the caller flattens higher-rank weights to the (rows, cols) layout the consuming op expects:
• upload(view, rows, cols, dst) — dtype-preserving: dst gets the brotensor dtype matching the view, zero conversion (a BF16 view yields a BF16 tensor, an F16 view an FP16 tensor);
• upload_fp16(view, rows, cols, dst) — always FP16, converting host-side from F32 if needed;
• upload_as(view, rows, cols, want, dst) — at an explicit arithmetic dtype, converting host-side. Lets one module pick a compute dtype different from the global one — e.g. Flux runs BF16 on a pipeline whose dtype is FP16, because its activations overflow FP16;
• upload_compute(view, rows, cols, dst) — at compute_dtype() for the current default device, so one checkpoint serves either backend (BF16 widens to FP32 on CPU, narrows to FP16 on a GPU);
• upload_compute_checked(view, rows, cols, dst, name) — same, but first validates the view's dtype and element count, throwing tagged with the caller-supplied name and the safetensors key.
• write_file(path, entries) — write a .safetensors file from host data. Each WriteEntry carries name / dtype / shape / host pointer / byte count; dtype defaults to F16.
• Supported on-disk dtypes: F32, F16, BF16, I32, I64, U8, BOOL.

GGUF (<brotensor/gguf.h>)

mmap'd reader for GGUF model files. Namespace brotensor::gguf.

• File — opens and mmaps a .gguf file, parses header + metadata, exposes tensors as TensorInfo (name, GGUF type, mapped brotensor dtype, dtype_supported, shape, numel, raw data span). Also find_tensor() / get_tensor() / tensors(), version(), alignment(), tensor_count().
• Metadata: find_meta(key) / get_meta(key) / metadata(), returning GGUF Value / ValueType.
• shape_to_2d(shape) — collapse a GGUF n-d shape to brotensor's (rows, cols). GGUF shapes are innermost-first, so cols = shape[0] and rows = product(shape[1..]); a 1-D shape gives (shape[0], 1). Throws on an empty shape.
• upload_raw(info, rows, cols, dst) — upload a tensor's raw bytes to the device at its carrier dtype, no host-side dequantization. Throws if info.dtype_supported is false; for a quant carrier, cols must be a multiple of dtype_block_size(dtype).
• Carriers the reader maps: F32, F16, BF16, the legacy blocks Q4_0 / Q4_1 / Q5_0 / Q5_1 / Q8_0 / Q8_1, and the K-quant superblocks Q2_K / Q3_K / Q4_K / Q5_K / Q6_K / Q8_K.

Note that carrier support is broader than op support: the reader will load any of the above, but only Q4_K / Q6_K / Q8_0 are consumed by the fused dequant / matmul kernels (see op-coverage.md). Loading a Q5_K tensor succeeds; calling a matmul on it throws.