A generative-AI field guide · source material 2025 · updated 2026

How electricitybecomes intelligence.

The large-language-model and generative-AI industry is not a model, chip, or app. It is a coupled production system—from electrons and silicon to tokens and work.

9 layers3 source perspectives1 economic flywheel

MATRIX MULTIPLY

electrons → floating-point operations → tokens → work

Follow one request

01 A simple question

“Research this market, test the thesis, and brief me tomorrow.”

background agent running

The interface hides the factory

A sentence crosses nine industries before it becomes an answer.

The request is packaged by an application, interpreted by a model, scheduled by software, moved through memory, multiplied across silicon, routed over networks, and paid for in watts. The polished response at the top rests on a physical supply chain at the bottom.

MARKETSWhy Nvidia’s profit pool attracts attackN1

MODELS + GEOPOLITICSWhy DeepSeek changed the cost debateN2

SYSTEMSWhy background agents invert infrastructure choicesN3

The whole machine

Nine layers. No layer wins alone.

Value moves upward; constraints travel downward. A change in workload at the top can reorder the preferred software, chip, data center, and power source below it.

01ApplicationsAgents, copilots, tools 02ModelsWeights, attention, reasoning 03Data & learningCorpora, feedback, RL worlds 04RuntimeKernels, compilers, schedulers 05Compute systemsMemory, compute, interconnect 06SiliconGPUs, custom chips, accelerators 07Supply chainFabs, HBM, packaging 08Data centersClouds, racks, fiber, uptime 09Physical inputsPower, land, water, cooling

The first fork

Are we building the model—or running it?

“AI compute” hides two different factories. Training creates and changes the weights. Inference uses those weights to answer requests and take actions.

BUILD THE MODELPre-train → post-train

Learn broad patterns, then shape behavior with supervised examples, preferences, safety work, and reinforcement learning.

RUN THE MODELPrefill → decode → act

Process context, generate tokens, search or verify at inference time, and turn the result into a useful outcome.

Within inference, the waiting tolerance changes the machine.

A person is waiting

Optimize every visible second.

Requests arrive unpredictably and users notice pauses. The system reserves headroom, spreads one model across fast-connected GPUs, and accepts lower utilization to deliver the next token quickly.

North star: Latency
System shape: Narrow + fast
Hard constraint: Interconnect and tail latency
Wait tolerance: Near-continuous service

The bus and the car

Latency gives one passenger a direct ride. Throughput fills every seat.

GPUs are happiest with large batches of parallel work. Interactive chat often sacrifices utilization to keep one user moving. Background agents can wait for a full batch—the foundation of the “token factory” thesis. N3

The research request begins its journey

Every useful answer is a relay race.

The same request now follows us through the stack—from intent to infrastructure and back to a verified brief.

01Intent

The app assembles the goal, context, tools, and rules.

02Encode

Text becomes tokens and the scheduler admits the request.

03Prefill

The model processes the prompt in parallel and creates state.

04Recall

Weights and key-value state stream through memory.

05Multiply

Accelerators execute the model’s kernels layer by layer.

06Coordinate

Devices exchange partial results across the fabric.

07Decode

A probability distribution yields a token, then repeats.

08Act

The harness verifies, calls tools, or continues reasoning.

At every step: power, cooling, supply, and capital keep the machine alive.

Anatomy of the stack

From intent to infrastructure.

Applications & agents

Turn probability into accountable work.

The model only predicts tokens. The application supplies the goal, context, tools, permissions, memory, stopping rule, and user experience that make those tokens useful.

THE REQUEST The brief becomes a plan: gather evidence, call tools, check sources, and stop only when the requirements are met.

MODELproposes

contexttoolsmemory

permissionsevalsretries

verified outcome

THE MOATWorkflow integration and proprietary context

THE FAILUREError compounds across long chains of actions

THE HUMAN EDGETaste, judgment, accountability, and exception handling

“Agent” should mean a system that adapts and completes an open-ended task—not a scripted chain wearing an AI label. Reliability is multiplicative: 99% success at each of 100 steps yields only 36.6% end-to-end success. N4

Models

Compress patterns; spend compute to recover them.

A transformer alternates two different jobs. Attention decides which prior tokens matter now; the feed-forward network transforms that signal through learned weights. During low-batch decoding, memory movement often dominates. Larger batches and prompt processing can shift the bottleneck back toward math.

THE REQUEST The model turns the brief, retrieved evidence, and its learned patterns into candidate next tokens.

thenextusefultoken

ATTENTION

Which parts of the context matter?

memory movement · KV cache

× many layers

FEED-FORWARD / EXPERTS

What transformation should happen?

matrix multiplication · learned weights

probability → sampled token

01Pre-training

Predict the next token across vast corpora. Produces a base model.

02Post-training

Shape behavior with supervised fine-tuning, preferences, safety work, and reinforcement learning.

03Inference-time reasoning

Spend more tokens on search, verification, tool use, or alternative solution paths for this answer.

Training changes the model. Inference-time reasoning does not; it spends more compute on a particular question after training is complete. N5

Data & learning

The internet was the first subsidy. Environments are the next.

Human text built general language models, but high-quality public data is finite. Progress increasingly comes from curated expert data, synthetic examples, and environments where an answer can be checked.

THE REQUEST Its quality depends on what the model learned, what the agent retrieves, and whether the resulting claims can be checked.

WEB + BOOKS + CODEgeneral patterns

CURATIONquality and provenance

HUMAN / AI FEEDBACKdesired behavior

RL ENVIRONMENTSverifiable outcomes

VERIFIABLE

The system can grade itself.

Code that passes tests
Math with a proof or answer
Games and simulations
Structured computer actions

SOFT / CONTESTED

The target contains human judgment.

Taste and originality
Ambiguous strategy
Values and trade-offs
Open-world social outcomes

Open weights ≠ open source

Downloadable weights let you run a model privately. Full reproducibility also requires the training data, code, recipe, and license. DeepSeek widened access, but did not publish every ingredient. N6

Runtime & systems software

Make the expensive machine behave as if nothing is wasted.

Every abstraction trades convenience for control. The serving stack converts a model graph into kernels, assigns work to devices, batches users, manages caches, and recovers from failures.

THE REQUEST The runtime admits the job, selects kernels, batches compatible work, places caches, and recovers when a device fails.

PRODUCTAgent / API / applicationintent and tools

FRAMEWORKPyTorch · JAX · model librariesgraphs and tensors

COMPILERTriton · XLA · vendor toolchainslower and fuse operations

PLATFORMCUDA / NCCL · ROCm · custom runtimesdevice and collective control

KERNELMatrix multiply · attention · routingmicroseconds on silicon

Kernels

Fuse operations so intermediate results stay close to compute instead of taking a round trip to memory.

Batching

Combine requests to fill the GPU, trading individual latency for total throughput.

Parallelism

Split work by tensors, pipeline stages, experts, or requests—each with a different communication bill.

Control plane

Route, retry, checkpoint, and move work when hardware or a whole site disappears.

Nvidia’s moat is not a single language. It is the libraries, drivers, tools, documentation, trained labor, and predictable behavior around CUDA. Higher-level compilers can erode lock-in only if they preserve performance. N7

Compute systems

The scarce resource is often movement, not multiplication.

Tensor cores can perform staggering amounts of math. They still sit idle if weights, activations, or another GPU’s partial result do not arrive in time.

THE REQUEST Model weights and the growing key-value cache must reach compute fast enough to keep the brief moving.

ON-CHIP SRAMfastest · tiny

HBMfast · scarce · costly

HOST DRAMlarger · farther

NVMe / FLASHvast · much slower

CAPACITY ↑

SPEED ↑

THE KV CACHE

The model’s dynamic working memory: a representation of every prior token needed during decoding.

conversation lengthmemory pressure

COMPUTEHow many operations?

∩

MEMORYHow many bytes moved?

∩

INTERCONNECTHow quickly between chips?

→

THROUGHPUT CEILINGThe lowest ceiling sets the pace.

Useful throughput ≤ the lowest of the compute, memory, and communication ceilings.

Cerebras and Groq push more fast memory and deterministic execution to maximize token speed. GPUs remain unusually balanced. A likely endpoint is heterogeneous: different devices for attention, experts, prefill, or decode. N8

Silicon

There is no best chip—only a workload and a price.

A general-purpose GPU balances math, memory, programmability, and networking. Specialized accelerators deliberately become “spiky”: exceptional on one axis, constrained on another.

THE REQUEST Its latency target, batch size, context, model shape, and software decide which accelerator is economical.

Balanced GPU

Broad models, training, inference, mature tooling, fast scale-up networking.

Nvidia · AMDStrength: flexibility + ecosystem

Hyperscaler custom silicon

Hardware and software co-designed around the owner’s models and fleet.

Google TPU · AWS Trainium / Inferentia · Microsoft Maia · Meta MTIAStrength: captive scale + lower markup

Inference specialist

Wafer-scale, SRAM-heavy, deterministic, or architecture-specific bets.

Cerebras · Groq · Etched · d-MatrixStrength: extreme performance on a narrow path

THE REAL SCORECARD

$/useful tokennot $/theoretical FLOP

tokens / wattat the target batch and context

time to deployincluding software and debugging

supply at scaleincluding HBM and packaging

Fabrication & supply

A chip design is a promise. The supply chain makes it physical.

Frontier accelerators depend on a small set of deeply specialized producers. A shortage in wafers, high-bandwidth memory, packaging, substrates, or optics can strand everything else.

THE REQUEST The available machine was determined years earlier by design tools, wafer starts, memory orders, packaging capacity, and optics.

01TOOLS

ASML + equipment ecosystem

lithography, deposition, etch, metrology

02LOGIC FAB

TSMC · Samsung · Intel

turn designs into silicon dies

03MEMORY

SK hynix · Samsung · Micron

stack high-bandwidth memory (HBM)

04PACKAGING

CoWoS and peers

join logic, HBM, and interposers

05SYSTEM

boards · racks · optics

deliver a programmable machine

WHY TSMC MATTERS

More chip designers; fewer frontier manufacturers.

The foundry model pools demand across customers and amortizes extraordinary R&D and fab cost. Advanced packaging is now part of the performance story because it physically integrates logic with HBM. N9

leading-edge wafersHBMadvanced packagingpower

Data centers & cloud

Training wants a fortress. Inference can become a network.

Training synchronizes many accelerators on one fragile, tightly coupled job. Most inference requests are independent and can be spread across regions—especially when no human is waiting.

THE REQUEST Because the brief is due tomorrow, the job can be batched, retried, or moved instead of reserving the fastest possible path.

THE FORTRESS

Mega-cluster

Thousands of accelerators, one high-bandwidth fabric, enormous power density.

frontier pre-training
large-scale RL / self-play
low-latency model parallelism

THE FEDERATION

Distributed fleet

Many smaller sites, mixed chips, independent queues, software-defined resilience.

batched inference
background agents
interruptible workloads

DIMENSIONTRAININGINFERENCE

Job shape

One giant synchronized run

Many requests and agents

Network

Fast fabric is existential

Varies by model and latency target

Location

Concentrated

Can be geographically distributed

Failure

Can disrupt the whole run

Route, retry, or resume elsewhere

Economics

Speculative capability investment

Usage tied to delivered service

Movva’s “scavenger” strategy buys chips, sites, and uptime others reject, then uses software to pool them. The wager is not that unreliable infrastructure is good; it is that asynchronous work can absorb unreliability at the right price. N10

Power, land & cooling

The final input is not data. It is electricity, continuously delivered.

Servers convert electricity into heat and computation. The site must secure generation, transmission, substations, fiber, cooling, land, permits, and a path to reject that heat—at the same time.

THE REQUEST Every token in the finished brief becomes heat that must be powered, cooled, financed, and delivered reliably.

GENERATIONgas · nuclear · wind · solar

GRIDtransmission · substation

RACKpower distribution · compute

HEATliquid loops · chillers

STEADY + DENSE

Train where power never blinks.

Expensive accelerators make idling costly. Large clusters favor firm generation, redundant delivery, and liquid cooling.

CHEAP + FLEXIBLE

Move patient work to spare electrons.

Interruptible inference can follow weather, curtailment, or underused small sites—if the control plane can move the work.

A constraint across every layer

Policy changes the machine builders are allowed to assemble.

Export controls, model licenses, data rules, and geographic concentration do not sit beside the stack. They change the inputs available to every layer inside it.

01 · RULETrade and data policy

Governments define which chips, models, data, and services may cross a border.

02 · MACHINEAvailable specifications

Limits on interconnect or performance alter the hardware a team can buy.

03 · RESPONSEArchitecture and software

Builders compensate with routing, communication overlap, memory efficiency, and scale-out design.

04 · EXPOSUREConcentrated supply

Leading-edge fabrication and packaging remain geographically concentrated, while sovereignty pushes demand outward.

The H800 and H20 illustrate the loop: policy changed available specifications, then engineering adapted. This is a systems mechanism, not a geopolitical forecast. N14

CASE
STUDY
01

DeepSeek

Efficiency is a stack of small victories.

DeepSeek-V3 did not discover one magic shortcut. It combined architecture, numerical precision, routing, communication, low-level software, and training discipline. R1 post-training and open distribution are separate parts of the story.

2.788MH800 GPU-hours reported for V3’s full training

The widely repeated cost figure covers the reported V3 training run—not every prior experiment, salary, cluster purchase, R1 post-training step, or serving cost. That boundary is essential to an honest comparison. N11

V3 architecture

Choices inside the model. The blocks below are deliberately unscaled; they are not measured contribution percentages.

MIXTURE OF EXPERTS

Activate a small subset of the model for each token.

MULTI-HEAD LATENT ATTENTION

Compress attention state and reduce key-value-cache memory.

MULTI-TOKEN PREDICTION

Train the model to predict several future tokens.

LOAD BALANCING

Route tokens across experts without a conventional auxiliary loss.

Training systems

Methods that make the training run fit and keep the cluster productive.

8-BIT FLOATING POINT

Use lower numerical precision while protecting sensitive calculations.

COMMUNICATION OVERLAP

Overlap computation and traffic, with lower-level scheduling around constrained links.

Post-training and diffusion

What changes behavior after V3 training—and how capability spreads afterward.

REINFORCEMENT LEARNING

Learn reasoning behavior from verifiable rewards during post-training.

DISTILLATION

Transfer behavior from a stronger model into a smaller one.

OPEN WEIGHTS

Let others run, adapt, and study the model; this is distribution, not a training efficiency.

THE TECHNICAL LESSON

Constraints can force full-stack co-design.

Less interconnect headroom made communication scheduling and architecture more important, not less.

THE ECONOMIC LESSON

Cheaper capability does not settle total demand.

It can reduce hardware per task while creating far more tasks—the Jevons-paradox counterargument.

THE STRATEGIC LESSON

Capability diffuses faster than capital stock.

Weights and papers travel instantly. Fabs, HBM, data centers, and grid connections do not.

The argument at the center

Nvidia can remain indispensable—and still face pressure.

The sources disagree because they ask different questions. Technical leadership, total compute demand, market share, and investment returns are related, but they are not the same variable.

THE DURABILITY CASE

The moat is a working system.

CUDA + libraries: developers inherit years of reliable, optimized software.
Nvidia Collective Communications Library + NVLink: fast collectives keep tightly coupled workloads productive.
Rack-level integration: chips, switches, CPUs, cooling assumptions, and support arrive together.
Cadence + trust: buyers know the roadmap, talent pool, and deployment path.
Demand elasticity: lower cost can unlock more inference than efficiency removes.

N12

THE COMPRESSION CASE

Every supernormal profit invites a route around.

Customer silicon: Google, AWS, Microsoft, Meta, and others can avoid a supplier’s markup.
Software abstraction: Triton, JAX/XLA, AI-written kernels, and better ROCm can weaken CUDA lock-in.
Inference fragmentation: specialist chips can win narrow but enormous workloads.
Algorithmic efficiency: MLA, MoE, quantization, distillation, and better scheduling reduce compute per task.
Open models: capability and pricing power diffuse beyond closed frontier labs.

N13

THE SYNTHESIS

Nvidia’s deepest moat is coordination: a product that works across silicon, memory, networking, software, and support. Its deepest threat is not “a faster chip.” It is a world where workloads split into enough stable categories that customers can replace a balanced platform with cheaper specialists. Efficiency can expand the market and compress the profit per unit at the same time.

Who plays where

The value chain—and each layer’s fault line.

No company owns the entire path. Many are trying to own one layer above or below their current one.

LayerRepresentative playersWhat is soldCore tension

Experiences

OpenAI · Anthropic · Microsoft · Google · Apple · Cursor · vertical AI

Outcomes, attention, distribution

Trust, workflow fit, agent reliability

Models

OpenAI · Anthropic · Google DeepMind · Meta · xAI · DeepSeek · Mistral

Capability per dollar

Frontier premium vs. open diffusion

Cloud & inference

AWS · Azure · Google Cloud · Oracle · CoreWeave · Together · Fireworks · Baseten · Sail

GPU-hours and tokens

Latency, throughput, utilization

Networking & optics

Nvidia / Mellanox · Broadcom · Marvell · Arista · Cisco · optical suppliers

Bandwidth, latency, ports, fiber

Scale-up performance vs. open scale-out fabrics

Systems software

CUDA / NCCL · ROCm · PyTorch · Triton · JAX / XLA · serving engines

Portability and utilization

Ecosystem lock-in vs. abstraction

Accelerators

Nvidia · AMD · Google TPU · AWS Trainium · Microsoft Maia · Meta MTIA · Cerebras · Groq · Etched

FLOPs, bytes, watts, yield

Balanced generalist vs. spiky specialist

EDA & chip IP

Synopsys · Cadence · Arm · verification and interface IP suppliers

Design tools, cores, interfaces

Faster design cycles vs. concentrated dependencies

Memory & packaging

SK hynix · Samsung · Micron · TSMC · ASE · Amkor · substrate suppliers

HBM stacks and packaged systems

Capacity, thermals, yield, long lead times

Manufacturing

TSMC · Samsung · Intel · ASML · Applied Materials · Lam Research · KLA

Wafers, HBM, packaged systems

Concentrated capacity and long lead times

Physical plant

Data-center developers · utilities · grid operators · Vertiv · Schneider Electric · Eaton

Megawatts, uptime, land, permits

Concentration vs. stranded capacity

The economic machine

Two loops run at once.

LOOP A · EXPANSION

CHEAPER
TOKENS

NEW
USES

MORE
DEMAND

MORE
CAPACITY

Efficiency expands the possible.

When intelligence becomes 10× cheaper, developers stop rationing it and invent workloads that were previously irrational.

LOOP B · COMPRESSION

BETTER
MODELS

FEWER
RESOURCES

MORE
SUPPLIERS

LOWER
MARGINS

Efficiency attacks the incumbent price.

Custom chips, open models, and portable software can reduce the amount paid to any one layer—even as the total market grows.

THE BRIDGE FROM DEMAND TO RETURNS

Useful work must pay for the whole machine.

Growing token demand does not determine which layer captures profit. Price, utilization, asset life, and operating cost decide the return.

REVENUE

paid tokens, seats, or completed outcomes

−

CAPITAL

depreciation, financing, idle capacity

−

OPERATIONS

power, network, cooling, labor

RETURN

the profit retained by this layer

Higher utilization spreads fixed cost over more useful output. Rapid hardware obsolescence can shorten the period available to recover that cost. N15

CLOSED FRONTIER

A temporary capability lead.

Frontier labs can charge for superior quality, reliability, tools, and enterprise trust while the lead is meaningful.

OPEN DIFFUSION

A persistent path to cheaper capability.

Open weights, distillation, and public model outputs spread behavior. Enterprises still switch slowly, so closed and open models can coexist.

VERTICAL FUTURE

Own the model, cloud, chips, and distribution.

Best for giant labs and hyperscalers that can co-design the stack and finance frontier clusters.

MARKET FUTURE

Mix open models, rented silicon, and distributed power.

Best for providers that arbitrage heterogeneous supply and sell a clean token or outcome API.

The likely answer is both—segmented by workload.

Completing the map

Five pieces the infrastructure story can hide.

These are not extra layers for their own sake. They are what turns raw capability into durable, safe, economically useful systems.

Evaluation + security

Benchmarks, adversarial tests, monitoring, sandboxing, and incident response decide whether autonomous work can be trusted.

Persistent memory + retrieval

The KV cache is temporary working memory. Databases, object stores, search, and retrieval systems give agents durable organizational context.

Rights + provenance

Who owns training data, generated outputs, and user context shapes licensing, privacy, sovereignty, and which models enterprises can deploy.

Edge + on-device AI

Phones, PCs, vehicles, and robots trade model size for privacy, immediacy, lower cloud cost, and operation without a network.

Distribution + return on investment

Capability has no economic value until it reaches a workflow where saved time, reduced risk, or new revenue exceeds the full system cost.

The durable insight

The stack is not a ladder.
It is a moving bottleneck.

By morning, the agent can deliver the brief—but only if every layer supplied enough accuracy, throughput, reliability, and cost discipline.

Today’s winner is not automatically tomorrow’s. As software improves, scarcity moves to memory. As memory expands, it moves to packaging or power. As tokens get cheap, it moves to trust, distribution, and good questions. Understanding AI means watching where the bottleneck goes next—and which player can follow it across layers.

Return to the top