Skip to content

Mapping

The mapping ties a workload to a hardware system: it says which cores each operator may run on and how its work is split across them. The constraint-optimization (CO) pipeline then uses this as the search space and decides the concrete tensor placement and routing.

There are two things to keep distinct:

  • The spatial mapping (dataflow) - how a single operator is unrolled across a core's MAC array - lives on the core, in the hardware file (operational_array.dimensions / sizes). It is a property of the core, not of the mapping file.
  • The mapping file decides core allocation, inter-core tiling (splitting an operator across multiple cores), and intra-core tiling / fusion (how operators group and tile within a core).

A mapping can be auto-generated by the pipeline or hand-written. The format is validated by stream/parser/mapping_validator.py.


Auto-generated mapping (the default)

If you don't pass a mapping, or pass a placement file, evaluate_mapping asks the mapping generator that claims the hardware for one; for any accelerator that is the generic generator. This is the recommended starting point.

The generic generator (stream/stages/generation/generic_mapping_generation.py):

  • Selects eligible cores per node from the hardware. A core with an operator_types list only accepts those op types (so pooling goes to the pooling core, SiLU/Mul to the SIMD core); a core without the list accepts anything. A core listing an operator also runs the operators it is expanded into and its parts (a core listing Softmax runs PartialSoftmax). A node no core runs is refused, naming its operator. Memory cores are never chosen for computation.
  • Splits work across cores when several compute cores are eligible, by factoring the node's dimensions into an inter-core tiling.
  • Follows the placements of a placement file: it splits the cores first along the dimension indexing the axis the placements split the most bits along, so each core reads the block of the off-chip memory beside it; an operator reading what such a split made, in its group or across a fusion cut, follows that split in turn, so the data stays where it was made. It keeps the tiler from walking that dimension over time, and carries the placements into each group's mapping. A copy is local to every core, so it does not steer the split; for decode attention the generator may then split the sequence and combine the softmax across slices. To keep a copied cache split by heads, split its placement by heads or write the split into the mapping.
  • Splits attention along its sequence where it must (FlashDecodingStage). Attention is found by its structure: a contraction making scores along an axis only the keys index, element-wise ops on them, a softmax over that axis, and a contraction reducing it against the values. The sequence is split into parts, each part's softmax kept whole on its cores, and the parts are combined; an input only the attention reads along the sequence gets a part axis in place of the sequence's, so a placement splitting the sequence splits the parts. Attention that fits as the generator cuts it is left whole unless a placement splits its sequence. Otherwise the part count is the fewest that fit, a power of two times the memories a placement splits the sequence over, or that many memories where none fit. SolveOptions(stage_options={"attention_parts": n}) sets it; 1 leaves attention as it is.
  • Orders each group's tile loops (the intra_core_tiling list, innermost first) by the data they move off chip, taking another order than the default (from the dim indexing the least data to the most) only where it moves less off chip in less time, the time being the most data a memory takes over the bandwidth of the links bringing it from off chip or from a staging memory, each counted whole where several share a port (a memory without such links may take no more than before); a reduction whose output the group reads on stays inside the other loops. An intra_core_tiling that is given (the stage option or a mapping file) keeps its order, and groups tiling more than four dims keep the default. Given as the stage option, each group takes the entries naming its nodes, and a group no entry names is tiled as if none were given, to fit its memories.
  • Writes per-fusion-group mapping.yaml files under the run's output directory, so you can inspect (and later hand-edit) exactly what it chose.

This is enough to run any of the example workloads on any of the example architectures - see the matrix in the README.


Hand-written mapping

A mapping file is a list of layer entries (optionally wrapped in layers:), with optional fused_groups:.

layers:
  - name: Conv
    core_allocation:
      - [0, 1, 2, 3]          # candidate cores for Conv nodes
    inter_core_tiling:
      - - dim: D6
          split: 4            # split dimension D6 across 4 cores

  - name: Gemm
    core_allocation:
      - [0, 1, 2, 3]
    inter_core_tiling:
      - - dim: D2
          split: 4

  - name: MaxPool
    core_allocation:
      - [4]                   # the pooling core

  - name: Add
    core_allocation:
      - [5]                   # the SIMD core

fused_groups:
  - name: Fused_Group_1
    layers: [Conv]
    intra_core_tiling:
      - dim: Conv.D0
        tile: 1

Matching entries to nodes

For each node in the workload, Stream looks for a mapping entry in this order:

  1. Exact name - the entry name equals the node's name (e.g. Gemm_Left).
  2. Operator type - the entry name equals the node's op type (e.g. Gemm, Conv, Add). This is the common case, letting one entry cover all nodes of a type.

If neither matches, validation fails - every node must resolve to an entry (use a type-level entry to catch the rest).

Layer fields

Field Required Meaning
name yes Node name or operator type to match (see above).
core_allocation yes A list of candidate core-id groups. [[0,1,2,3]] is one group of four cores; the MILP allocator chooses the actual placement within that candidate set. A single-core role is just [[4]]. An empty list [] asks Stream to place the layer on the accelerator's core grid; a layer still without cores after that, or with an empty group [[]], is refused by name.
inter_core_tiling no How to split the operator across cores. Each inner entry is {dim: D<n>, split: k} - split loop dimension D<n> (0-indexed in the node's loop nest) by factor k.
kernel no Kernel used by the AIE codegen path: {name: <kernel>, kwargs: {<field>: <value>}}, described by the accelerator's kernel library. Ignored by the non-AIE CO pipeline.

Fused-group fields

fused_groups declares which layers are scheduled together as one fusion group and how they tile within a core.

Field Required Meaning
name yes Group label.
layers yes Names of the layers in this group.
intra_core_tiling no Per-dimension temporal tiling, each {dim: <Node>.D<n>, tile: size} - note the fully-qualified dimension name (e.g. Conv.D0). The list order is the loop order, innermost first, kept as given.
post_ops no Elementwise layers of the group that run as post-ops of the GEMM producing their input (AIE only): each folds into that GEMM, which writes the layer's output and, once its sum over a tile is done, calls the layer's kernel (silu, eltwise_mul) on its own cores at its output tile. A layer folds where it alone reads that GEMM's output, at the index it writes its own; a binary one folds into the later of the GEMMs it reads and reads its other input at that GEMM's output tile, from the core beside it when a GEMM of the group produces it. Every post-op and the GEMM it folds into are layers of the group.

Placement of inputs and outputs

Under runtime_args, a workload input or output has two separate properties:

  • its placement: which memories hold the tensor, and how it is split over them: off-chip memories, or on-chip ones where it stays for the whole run (see On chip below);
  • its layout: the order of the tensor's bytes inside a memory, the affine map the code generators read (see Data layout).

An accelerator may have several off-chip memories (see Hardware). A placement is optional: the mapping file may fix one per edge under runtime_args, keyed by the tensor's name, beside any layout, and the allocation chooses the rest (see Without a placement below):

runtime_args:
  K: {placement: {memories: [hbm_0, hbm_1, hbm_2, hbm_3], split: 1}}   # block s along axis 1 in hbm_s
  q: {placement: {memories: [hbm_0, hbm_1, hbm_2, hbm_3]}}             # a copy in each
  O: {placement: {memories: [hbm_0]}, layout: "(d0, d1) -> (d1, d0)"}  # all of it in one, transposed there
Placement Meaning
split: <axis> The whole tensor cut into one contiguous block per memory along that axis (0-indexed, as the workload declares the tensor), block s in memories[s]. The axis must split evenly.
several memories, no split A copy of the whole tensor in each; every reader takes it from the nearest. Inputs only.
one memory All of it there, as with a single off-chip memory.

A mapping file with only runtime_args placing the inputs and outputs is a placement file: the generic generator maps the rest (see Auto-generated mapping).

  • Without a placement. With several off-chip memories, the allocation chooses where an input without one lives: all of it in one memory, a block in each along any declared axis that splits evenly, or a copy in each, each lowered into its own routes, ports and links. The solve takes the fastest and, of placements as fast, the one holding the fewest bits off chip (the objective level after the latency), so it copies only where a copy is faster. Off-chip traffic is the same for every choice; energy is not an objective level. The mapping generator cuts the work by compute alone, its tiler keeping from walking the dim the cores split such an input along; the allocation then places the input where that cut reads it best. An input the user places is fixed. The first fused group reading a tensor chooses, and later groups read it there; the AllocationIR's edge_placements gives every input's placement, chosen or declared. An output without one lands a block per producing core in the off-chip memory the fewest hops from that core. The input end of a fusion cut without one reads it back from there, and is refused where one block lands in several memories.
  • Fusion cuts. The two ends of a cut, <node>_cut_out and <node>_cut_in, are one tensor: a placement declared for either holds for both, and ends declaring different placements are refused.
  • On chip. A placement may name on-chip memories instead, such as weights kept on chip between runs: w: {placement: {memories: [l2]}}. Its memories are all on chip or all off chip; one mixing them is refused. The tensor is there when the run starts and stays there to its end: it is never read from or written to an off-chip memory, and is no part of the off-chip footprint (offchip_holdings). Each reader reads it from that memory, the transfer starting there and priced by that memory's ports and the links it crosses, with no staging copy in a memory tile; an output, or the output end of a fusion cut, placed on chip lands there, and a later group reads it there. Its share of the whole tensor takes room in that memory in every fused group of the run, not only those reading it: before solving, the tensors placed in a memory must fit its capacity, else the run is refused as off chip, e.g. l2 holds w1 + w2 = 2 MiB > 1.5 MiB; inside each group the memory has that much less room for the group's own tensors, and a copy of a placed tensor in its own memory takes none. The mapping generator cuts the tiles to fit in that room too, and orders the tile loops counting the tensor as moving nothing off chip. The allocation never places a tensor on chip by itself: an input without a placement lives off chip as above. The AIE code generator, whose runtime sequence brings every input in from off chip, refuses on-chip placements.
  • Time and traffic. Each memory serves what its blocks hold of what each core reads. Where the iterations walk a split axis, one iteration may read from fewer memories than the walk does: the latency takes the busiest iteration, which waits on the memories it reads while the others idle, while traffic and energy take the average over the walk, each memory its share. The busiest iteration is an upper bound: a schedule prefetching more than one tile ahead may beat it.
  • Capacity. Before solving, each memory must hold what is placed in it: a block of each tensor split over it, all of each copy, each tensor counted once over the fused groups and both ends of a cut. Otherwise the run is refused, naming the memory and the sum, e.g. hbm_0 holds K + V + q/4 + O/4 = 32 MiB > 16 MiB. Where the allocation chooses an input's placement, each memory's capacity is a constraint of the solve: what the chosen placements hold there and what is placed there already must fit, so a copy or a pooled placement that overflows is never chosen. A later group's input the allocation places is reserved a block in each memory meanwhile, the least room it takes, so an earlier group's choice cannot leave it none. The AllocationIR's offchip_holdings gives, per off-chip memory, the bits of each workload tensor the group holds there, keyed by the tensor edge_tensors names: summed over the groups, each tensor once, they are the workload's footprint.

A placement the cores' split follows keeps each core's data in its own slice. Split along an axis the cores do not split, the data crosses the on-chip links; split along an axis the iterations walk, each iteration waits on one memory. tests/test_decode_kv.py shows all three for decode attention (stream/inputs/testing/workload/make_decode_attention.py) on stream/inputs/testing/hardware/tpu_like_ring_hbm.yaml and tpu_v7_ironwood_hbm_per_chiplet.yaml.


How the mapping feeds the optimizer

core_allocation defines the candidate set, not a fixed assignment (unless a role has only one core). The MILP allocator (AllocationModel) then chooses, within those candidates, where each tensor lives and which links carry each transfer - minimizing latency subject to memory and bandwidth constraints. inter_core_tiling determines how many parallel pieces exist to place; fused_groups / intra_core_tiling determine what is co-scheduled and how it is temporally tiled on a core.

For a workload with multiple fusion groups, the pipeline runs the CO once per group (see Stages).


Reusing an allocation

The auto-generated mapping.yaml files written into a run's output directory are valid hand-written mappings. To pin a result, copy the generated mapping out, edit the core_allocation candidate sets down to the chosen cores, and pass it back as evaluate_mapping's mapping.