CUDA Streams: Overlap Compute and Transfer

Your GPU has two kinds of busy: compute (the streaming multiprocessors, SMs) and data movement (the DMA copy engines that move bytes between host and device). A kernel that reads its input from a cudaMemcpy that just finished leaves one of those idle almost the whole time. This guide shows you how to keep both busy: streams for queuing work, asynchronous copies that don’t block the host, and events for stitching independent queues together — ending in a benchmark of a copy/compute pipeline against the plain serial version.

Who this is for

You already write CUDA kernels: you know <<<grid, block>>>, cudaMalloc, cudaMemcpy, and cudaFree. You have never used streams, and you want the mental model plus working code, not a tour of the API surface.

Prerequisites (with versions)

• Linux, a CUDA-capable NVIDIA GPU with a copy engine (asyncEngineCount > 0 — everything discrete from ~2012 on; page 01 shows how to check)
• CUDA Toolkit 13.2 (this guide’s code was compiled with nvcc 13.2.86, -arch=sm_80). CUDA 12.x works unchanged — every API used here is identical in 12.9. If you are on a 2025-or-newer distro, use 13.x: CUDA 12.9 refuses gcc > 14 and clashes with new glibc headers (see the traps on page 01).
• No other dependencies: these are plain nvcc programs.

What you’ll be able to do after this guide

1. Create streams and explain exactly what the default stream does to your concurrency.
2. Copy data with cudaMemcpyAsync — and know why it silently stops being asynchronous when your host memory isn’t pinned.
3. Synchronize streams with events, and time device work properly.
4. Benchmark a chunked, event-pipelined copy+compute loop against the serial version on your own GPU, and tune it.

60-second check

nvcc --version

Real output, from the machine this guide was built on:

Cuda compilation tools, release 13.2, V13.2.86
Build cuda_13.2.r13.2/compiler.37953736_0

The compiler alone is not enough — you also need the NVIDIA driver on the machine that will run the programs. This guide’s build box has the toolkit but no driver and no GPU, so running any of these programs there produces a real error you’ll learn to recognize on page 01 (and which is documented there).

Status of this guide

status: untested — every program on every page compiles cleanly with nvcc 13.2.86 (verified on this site’s build host), and every API claim is verified against the CUDA 12.8/13.2 official documentation and headers. But no program was executed: the build box and deploy host have no NVIDIA GPU. Outputs shown as “expected” are labeled as such; the only real captured outputs are compile results, the error strings, and the no-driver failure. If you run these on a GPU, the numbers on page 04 are yours to fill in — that’s the point of the benchmark.

Page map

• 01 — Streams: what a stream is, creating and destroying them, the default stream trap.
• 02 — Async memcpy: cudaMemcpyAsync, and why pinned memory decides whether you get overlap or a silent lie.
• 03 — Events: recording, waiting, querying, timing.
• 04 — The overlap benchmark: serial vs naive-async vs pipelined, with the full code, tuning knobs, and when overlap doesn’t help.

Out of scope

Multi-GPU and peer-to-peer copies (that’s a separate guide). Managed memory / cudaMallocAsync, CUDA graphs, and stream priorities are mentioned only as pointers in further reading.