Mapping¶
The mapping ties a workload to a hardware system: it says which cores each operator may run on and how its work is split across them. The constraint-optimization (CO) pipeline then uses this as the search space and decides the concrete tensor placement and routing.
There are two things to keep distinct:
- The spatial mapping (dataflow) - how a single operator is unrolled across a core's MAC array - lives on the core, in the hardware file (
operational_array.dimensions/sizes). It is a property of the core, not of the mapping file. - The mapping file decides core allocation, inter-core tiling (splitting an operator across multiple cores), and intra-core tiling / fusion (how operators group and tile within a core).
A mapping can be auto-generated by the pipeline or hand-written. The format is validated by stream/parser/mapping_validator.py.
Auto-generated mapping (the default)¶
If you don't pass a mapping, or pass a placement file, evaluate_mapping asks the mapping generator that claims the hardware for one; for any accelerator that is the generic generator. This is the recommended starting point.
The generic generator (stream/stages/generation/generic_mapping_generation.py):
- Selects eligible cores per node from the hardware. A core with an
operator_typeslist only accepts those op types (so pooling goes to the pooling core, SiLU/Mul to the SIMD core); a core without the list accepts anything. A core listing an operator also runs the operators it is expanded into and its parts (a core listingSoftmaxrunsPartialSoftmax). A node no core runs is refused, naming its operator. Memory cores are never chosen for computation. - Splits work across cores when several compute cores are eligible, by factoring the node's dimensions into an inter-core tiling.
- Follows the placements of a placement file: it splits the cores first along the dimension indexing the axis the placements split the most bits along, so each core reads the block of the off-chip memory beside it; an operator reading what such a split made, in its group or across a fusion cut, follows that split in turn, so the data stays where it was made. It keeps the tiler from walking that dimension over time, and carries the placements into each group's mapping. A copy is local to every core, so it does not steer the split; for decode attention the generator may then split the sequence and combine the softmax across slices. To keep a copied cache split by heads, split its placement by heads or write the split into the mapping.
- Splits attention along its sequence where it must (
FlashDecodingStage). Attention is found by its structure: a contraction making scores along an axis only the keys index, element-wise ops on them, a softmax over that axis, and a contraction reducing it against the values. The sequence is split into parts, each part's softmax kept whole on its cores, and the parts are combined; an input only the attention reads along the sequence gets a part axis in place of the sequence's, so a placement splitting the sequence splits the parts. Attention that fits as the generator cuts it is left whole unless a placement splits its sequence. Otherwise the part count is the fewest that fit, a power of two times the memories a placement splits the sequence over, or that many memories where none fit.SolveOptions(stage_options={"attention_parts": n})sets it;1leaves attention as it is. - Orders each group's tile loops (the
intra_core_tilinglist, innermost first) by the data they move off chip, taking another order than the default (from the dim indexing the least data to the most) only where it moves less off chip in less time, the time being the most data a memory takes over the bandwidth of the links bringing it from off chip or from a staging memory, each counted whole where several share a port (a memory without such links may take no more than before); a reduction whose output the group reads on stays inside the other loops. Anintra_core_tilingthat is given (the stage option or a mapping file) keeps its order, and groups tiling more than four dims keep the default. Given as the stage option, each group takes the entries naming its nodes, and a group no entry names is tiled as if none were given, to fit its memories. - Writes per-fusion-group
mapping.yamlfiles under the run's output directory, so you can inspect (and later hand-edit) exactly what it chose.
This is enough to run any of the example workloads on any of the example architectures - see the matrix in the README.
Hand-written mapping¶
A mapping file is a list of layer entries (optionally wrapped in layers:), with optional fused_groups:.
layers:
- name: Conv
core_allocation:
- [0, 1, 2, 3] # candidate cores for Conv nodes
inter_core_tiling:
- - dim: D6
split: 4 # split dimension D6 across 4 cores
- name: Gemm
core_allocation:
- [0, 1, 2, 3]
inter_core_tiling:
- - dim: D2
split: 4
- name: MaxPool
core_allocation:
- [4] # the pooling core
- name: Add
core_allocation:
- [5] # the SIMD core
fused_groups:
- name: Fused_Group_1
layers: [Conv]
intra_core_tiling:
- dim: Conv.D0
tile: 1
Matching entries to nodes¶
For each node in the workload, Stream looks for a mapping entry in this order:
- Exact name - the entry
nameequals the node's name (e.g.Gemm_Left). - Operator type - the entry
nameequals the node's op type (e.g.Gemm,Conv,Add). This is the common case, letting one entry cover all nodes of a type.
If neither matches, validation fails - every node must resolve to an entry (use a type-level entry to catch the rest).
Layer fields¶
| Field | Required | Meaning |
|---|---|---|
name | yes | Node name or operator type to match (see above). |
core_allocation | yes | A list of candidate core-id groups. [[0,1,2,3]] is one group of four cores; the MILP allocator chooses the actual placement within that candidate set. A single-core role is just [[4]]. An empty list [] asks Stream to place the layer on the accelerator's core grid; a layer still without cores after that, or with an empty group [[]], is refused by name. |
inter_core_tiling | no | How to split the operator across cores. Each inner entry is {dim: D<n>, split: k} - split loop dimension D<n> (0-indexed in the node's loop nest) by factor k. |
kernel | no | Kernel used by the AIE codegen path: {name: <kernel>, kwargs: {<field>: <value>}}, described by the accelerator's kernel library. Ignored by the non-AIE CO pipeline. |
Fused-group fields¶
fused_groups declares which layers are scheduled together as one fusion group and how they tile within a core.
| Field | Required | Meaning |
|---|---|---|
name | yes | Group label. |
layers | yes | Names of the layers in this group. |
intra_core_tiling | no | Per-dimension temporal tiling, each {dim: <Node>.D<n>, tile: size} - note the fully-qualified dimension name (e.g. Conv.D0). The list order is the loop order, innermost first, kept as given. |
post_ops | no | Elementwise layers of the group that run as post-ops of the GEMM producing their input (AIE only): each folds into that GEMM, which writes the layer's output and, once its sum over a tile is done, calls the layer's kernel (silu, eltwise_mul) on its own cores at its output tile. A layer folds where it alone reads that GEMM's output, at the index it writes its own; a binary one folds into the later of the GEMMs it reads and reads its other input at that GEMM's output tile, from the core beside it when a GEMM of the group produces it. Every post-op and the GEMM it folds into are layers of the group. |
Placement of inputs and outputs¶
Under runtime_args, a workload input or output has two separate properties:
- its placement: which memories hold the tensor, and how it is split over them: off-chip memories, or on-chip ones where it stays for the whole run (see On chip below);
- its layout: the order of the tensor's bytes inside a memory, the affine map the code generators read (see Data layout).
An accelerator may have several off-chip memories (see Hardware). A placement is optional: the mapping file may fix one per edge under runtime_args, keyed by the tensor's name, beside any layout, and the allocation chooses the rest (see Without a placement below):
runtime_args:
K: {placement: {memories: [hbm_0, hbm_1, hbm_2, hbm_3], split: 1}} # block s along axis 1 in hbm_s
q: {placement: {memories: [hbm_0, hbm_1, hbm_2, hbm_3]}} # a copy in each
O: {placement: {memories: [hbm_0]}, layout: "(d0, d1) -> (d1, d0)"} # all of it in one, transposed there
| Placement | Meaning |
|---|---|
split: <axis> | The whole tensor cut into one contiguous block per memory along that axis (0-indexed, as the workload declares the tensor), block s in memories[s]. The axis must split evenly. |
several memories, no split | A copy of the whole tensor in each; every reader takes it from the nearest. Inputs only. |
| one memory | All of it there, as with a single off-chip memory. |
A mapping file with only runtime_args placing the inputs and outputs is a placement file: the generic generator maps the rest (see Auto-generated mapping).
- Without a placement. With several off-chip memories, the allocation chooses where an input without one lives: all of it in one memory, a block in each along any declared axis that splits evenly, or a copy in each, each lowered into its own routes, ports and links. The solve takes the fastest and, of placements as fast, the one holding the fewest bits off chip (the objective level after the latency), so it copies only where a copy is faster. Off-chip traffic is the same for every choice; energy is not an objective level. The mapping generator cuts the work by compute alone, its tiler keeping from walking the dim the cores split such an input along; the allocation then places the input where that cut reads it best. An input the user places is fixed. The first fused group reading a tensor chooses, and later groups read it there; the AllocationIR's
edge_placementsgives every input's placement, chosen or declared. An output without one lands a block per producing core in the off-chip memory the fewest hops from that core. The input end of a fusion cut without one reads it back from there, and is refused where one block lands in several memories. - Fusion cuts. The two ends of a cut,
<node>_cut_outand<node>_cut_in, are one tensor: a placement declared for either holds for both, and ends declaring different placements are refused. - On chip. A placement may name on-chip memories instead, such as weights kept on chip between runs:
w: {placement: {memories: [l2]}}. Its memories are all on chip or all off chip; one mixing them is refused. The tensor is there when the run starts and stays there to its end: it is never read from or written to an off-chip memory, and is no part of the off-chip footprint (offchip_holdings). Each reader reads it from that memory, the transfer starting there and priced by that memory's ports and the links it crosses, with no staging copy in a memory tile; an output, or the output end of a fusion cut, placed on chip lands there, and a later group reads it there. Its share of the whole tensor takes room in that memory in every fused group of the run, not only those reading it: before solving, the tensors placed in a memory must fit its capacity, else the run is refused as off chip, e.g.l2 holds w1 + w2 = 2 MiB > 1.5 MiB; inside each group the memory has that much less room for the group's own tensors, and a copy of a placed tensor in its own memory takes none. The mapping generator cuts the tiles to fit in that room too, and orders the tile loops counting the tensor as moving nothing off chip. The allocation never places a tensor on chip by itself: an input without a placement lives off chip as above. The AIE code generator, whose runtime sequence brings every input in from off chip, refuses on-chip placements. - Time and traffic. Each memory serves what its blocks hold of what each core reads. Where the iterations walk a split axis, one iteration may read from fewer memories than the walk does: the latency takes the busiest iteration, which waits on the memories it reads while the others idle, while traffic and energy take the average over the walk, each memory its share. The busiest iteration is an upper bound: a schedule prefetching more than one tile ahead may beat it.
- Capacity. Before solving, each memory must hold what is placed in it: a block of each tensor split over it, all of each copy, each tensor counted once over the fused groups and both ends of a cut. Otherwise the run is refused, naming the memory and the sum, e.g.
hbm_0 holds K + V + q/4 + O/4 = 32 MiB > 16 MiB. Where the allocation chooses an input's placement, each memory's capacity is a constraint of the solve: what the chosen placements hold there and what is placed there already must fit, so a copy or a pooled placement that overflows is never chosen. A later group's input the allocation places is reserved a block in each memory meanwhile, the least room it takes, so an earlier group's choice cannot leave it none. The AllocationIR'soffchip_holdingsgives, per off-chip memory, the bits of each workload tensor the group holds there, keyed by the tensoredge_tensorsnames: summed over the groups, each tensor once, they are the workload's footprint.
A placement the cores' split follows keeps each core's data in its own slice. Split along an axis the cores do not split, the data crosses the on-chip links; split along an axis the iterations walk, each iteration waits on one memory. tests/test_decode_kv.py shows all three for decode attention (stream/inputs/testing/workload/make_decode_attention.py) on stream/inputs/testing/hardware/tpu_like_ring_hbm.yaml and tpu_v7_ironwood_hbm_per_chiplet.yaml.
How the mapping feeds the optimizer¶
core_allocation defines the candidate set, not a fixed assignment (unless a role has only one core). The MILP allocator (AllocationModel) then chooses, within those candidates, where each tensor lives and which links carry each transfer - minimizing latency subject to memory and bandwidth constraints. inter_core_tiling determines how many parallel pieces exist to place; fused_groups / intra_core_tiling determine what is co-scheduled and how it is temporally tiled on a core.
For a workload with multiple fusion groups, the pipeline runs the CO once per group (see Stages).
Reusing an allocation¶
The auto-generated mapping.yaml files written into a run's output directory are valid hand-written mappings. To pin a result, copy the generated mapping out, edit the core_allocation candidate sets down to the chosen cores, and pass it back as evaluate_mapping's mapping.