Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attention

Sood, Aryan; Acharya, Shantanu; Nayak, Gaurav Kumar

Method Context

the host's own block anchor (copy of block 0) attention sink, 64 tokens summary of an earlier block, 512 tokens

Figure 1. Phase-1 input on each of four hosts, drawn to scale. Star Attention prepends a full copy of the first block to every later host. Pulsar Attention prepends a 64-token attention sink and a 512-token summary of each earlier block. Cost figures are per layer for the last (critical) host on Llama-3.1-8B-Instruct in bf16, from Table 6 of the paper.

Abstract

Inference with large language models (LLMs) on long sequences is computationally expensive due to the quadratic complexity of self-attention. Distributed blockwise methods such as Star Attention reduce this cost by sharding context across hosts, but rely on prepending a static, content-blind copy of the first block to every host. We propose Pulsar Attention, which replaces the static anchor with two lightweight, content-aware components: a small attention-sink prefix that stabilizes softmax, and compact cross-block summaries built via a Max-IDF heuristic that selects chunks containing globally rare tokens. This reduces the Phase 1 per-GPU FLOPs by up to 3.3× over Star Attention while retaining an identical KV cache footprint. On RULER with Llama-3.1-8B-Instruct, Pulsar Attention outperforms Star Attention at sequence lengths up to 128K tokens and remains competitive with dense attention across most tasks, with task-dependent absolute gains of up to 4.7% over the dense baseline.

1The anchor problem

Star Attention splits a long context into contiguous blocks and encodes them on separate hosts without any communication, then recovers global attention at query time with an online-softmax merge. Encoding blocks in isolation creates spurious attention sinks at the start of every block, so Star Attention prepends a copy of the first block, the anchor, to each host's input.

The anchor works, but it is the same for every host regardless of content. It doubles the per-host sequence length during encoding, and block i learns nothing about blocks 1 through i−1. As contexts get longer, the first block covers a shrinking share of what later blocks would need to see.

2Method

Pulsar Attention keeps Star Attention's two-phase structure and replaces only the anchor. Each host prepends two things to its block: the first 64 tokens of the sequence as an attention sink, and a short summary of every earlier block.

2.1 Selecting summaries with Max-IDF

Before any GPU work, every host builds the same inverse-document-frequency table from the token IDs of the full sequence, treating each block as a document. This is a single integer pass on the CPU and needs no communication. Each block is cut into non-overlapping 32-token chunks, and a chunk is scored by its rarest token:

where N is the number of blocks and df(t) the number of blocks containing t. The top-k chunks, kept in their original order, form the summary σ_i, with a budget of 12.5% of the block size. Host i then encodes with original position IDs, and discards the sink and summary KV entries afterwards.

We take the maximum rather than the mean because the tokens that matter for retrieval, such as identifiers, names and numbers, usually appear as a single rare token inside an otherwise ordinary chunk. Averaging divides that signal by the chunk length. Figure 2 shows the difference on a small example.

Selection query: What is the key?

Figure 2. A toy context of four blocks with 4-token chunks and one chunk kept per block. Filler sentences occur in every block (IDF 0, grey), a distractor phrase in two blocks (IDF log 2, light shading) and the answer orchid-4417 in one (IDF log 4, dark shading). The numbers are chunk scores; the outlined chunk is the one selected. Under each block is what that host prepends.

2.2 Query phase

After Phase 1 each host holds the KV cache of its own block only, so the retained cache is identical to Star Attention's. Phase 2 is unchanged: the query is broadcast, each host runs local attention over its shard, and an all_gather followed by an online-softmax merge gives exact attention over the retained shards. Pulsar can therefore be used anywhere Star Attention is, and combined with post-hoc KV compression.

3Results

All experiments use four hosts. Contexts of 16K to 64K are split into four blocks and 128K into eight, with two blocks per host. The main model is Llama-3.1-8B-Instruct, evaluated on all 13 RULER tasks with 500 samples per task and length.

Model include single-GPU baselines

Data

Figure 3. RULER average accuracy (%) against context length. Full attention runs on a single GPU; baseline numbers for Llama are from Acharya et al. (2025) with the same model and setup.

On Llama-3.1-8B, Pulsar is ahead of Star Attention at 32K (+2.8), 64K (+6.1) and 128K (+4.7), and ahead of full attention at the same lengths by +4.0, +4.7 and +2.8. At 16K it trails Star by 0.9 and full attention by 1.8 points. On Qwen3-4B, without any tuning for that architecture, it stays within about a point of full attention.

The average gain is not spread evenly across tasks. Figure 4 breaks the 128K result down by task.

Data

Figure 4. Per-task accuracy (%) at 128K on Llama-3.1-8B-Instruct. NIAH Single and MultiKey average three sub-tasks; QA averages SQuAD and HotpotQA.

Most of the gain comes from aggregation. On Common Words Extraction, full and Star Attention both fall to 0.04% at 128K, while Pulsar scores 81.2%; Frequent Words also improves, from 72.3% (full) to 80.7%. We think the summaries make evidence that repeats across blocks easier to aggregate, though our experiments do not establish this causally.

The clearest weakness is NIAH MultiValue, where Pulsar scores 63.5% against 91.6% for full attention. The task asks for several values stored under one key. Max-IDF finds the chunk with the rare key but can miss chunks holding values that are not globally rare.

3.1 Choice of scoring heuristic

Table 1 compares Max-IDF with three other statistical scores and two non-statistical baselines on BABILong at 16K. At the 12.5% budget used throughout the paper, Max-IDF gives the best results on the two multi-hop tasks, qa3 and qa5. It is not best everywhere: evenly spaced sampling wins on qa1, where broad positional coverage matters more than rare tokens.

Table 1. BABILong-16K accuracy (%), four blocks, σ = 512. Bold marks the best of the four statistical scores.

Table 2. InfiniteBench at 128K: summarization, book-level multiple choice and repository-level debugging.

4Efficiency

Each Pulsar host adds a fixed 1,600-token prefix (64 sink tokens plus three 512-token summaries) instead of a full block, so the saving grows with context length. We measured end-to-end latency and peak memory on one node with 4 × A100-80GB (NVLink), bf16 and FlashAttention-2, averaged over all RULER tasks.

Wall-clock speedup over full attention

Total GPU memory reduction over full attention

Figure 5. Left: end-to-end speedup per sample relative to full attention. Right: peak memory summed over all GPUs, relative to full attention. Star Attention is omitted on the right because its measured memory after KV discard is identical to Pulsar's.

Table 3. Phase-1 cost per layer on the critical host (Llama-3.1-8B-Instruct, four blocks, σ = 512, 64 sink tokens, bf16).

5Code

The implementation uses HuggingFace Transformers and works with LLMs trained with global attention, with no training or fine-tuning.

git clone https://github.com/zer0-data/pulsarattn.git
cd pulsarattn
pip install -r requirements.txt
python -c "import nltk; nltk.download('punkt_tab')"

To run on your own JSONL data with the settings from the paper (4K blocks, 16 chunks of 32 tokens):

torchrun --nproc_per_node=4 run_star_attn_inference.py \
  --model_path /path/to/Llama-3.1-8B-Instruct \
  --attn_type star --block_size 4096 \
  --summary_method max_idf --summary_chunks 16 \
  --chunk_size 32 --sink_size 64 \
  --tokens_to_generate 128 \
  --input_path data/input.jsonl --output_path results/output.jsonl

Launchers for RULER, BABILong and InfiniteBench are described in the repository README.

Citation

@inproceedings{sood2026pulsar,
  title     = {Dropping the Anchor: Statistical Context Summarization
               for Distributed Systems via Pulsar Attention},
  author    = {Sood, Aryan and Acharya, Shantanu and Nayak, Gaurav Kumar},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  url       = {https://arxiv.org/abs/2607.20457}
}

An earlier version was presented at the ICML 2026 Workshop on Resource-Adaptive Foundation Model Inference.

Acknowledgments

We acknowledge the National Supercomputing Mission (NSM) for providing computing resources of ‘PARAM Ganga’ at the Indian Institute of Technology Roorkee, which is implemented by C-DAC and supported by the Ministry of Electronics and Information Technology (MeitY) and Department of Science and Technology (DST), Government of India.