Pipeline Agents: Self-Hosted, Hybrid & VM Templates
Choose between Microsoft-hosted and self-hosted agents. Design agent infrastructure for cost, connectivity, licensing, and hybrid scenarios including VM templates and air-gapped environments.
What are pipeline agents and runners?
Think of construction workers on a building site.
The architect (your YAML pipeline) draws the blueprints, but someone has to actually hammer the nails and lay the bricks. Those are the construction workers — your agents.
You have two options: hire temporary workers from a staffing agency (Microsoft-hosted agents) — they show up clean every morning and leave at night, no maintenance needed. Or you hire permanent staff (self-hosted agents) — they know your building site, have specialized tools, and can access restricted areas, but you’re responsible for paying them and keeping their skills up to date.
Most teams use agency workers (hosted agents) for standard jobs and permanent staff (self-hosted) only when they need something the agency can’t provide — like access to a private network or specialized hardware.
Microsoft-hosted vs self-hosted
| Factor | Microsoft-Hosted Agents / GitHub-Hosted Runners | Self-Hosted Agents / Self-Hosted Runners |
|---|---|---|
| Provisioning | Fresh VM spun up per job — clean environment every time | Persistent machine — you install, configure, and maintain it |
| Maintenance | Zero — Microsoft/GitHub handles OS updates, tool updates, patching | Full responsibility — you manage OS patches, agent updates, tool installations |
| Cost | Free tier included (Azure: 1 parallel job, 1800 min/month; GitHub: 2000 min/month). Pay for extra parallelism. | No per-minute cost — but you pay for the VM/hardware, electricity, and maintenance time |
| Network access | Public internet only — cannot reach private VNets or on-prem resources without extra config | Can access private networks, on-prem databases, internal APIs, air-gapped environments |
| Performance | Standard VM sizes (2 vCPU, 7 GB RAM typical). GitHub Larger Runners offer up to 64 vCPU. | Any hardware you choose — GPU machines, high-memory VMs, dedicated build servers |
| Clean state | Guaranteed clean — VM is destroyed after each job (no leftover state) | Persistent — can cache dependencies, but risks stale state, leftover files, security residue |
| Software | Pre-installed tools (major SDKs, Docker, etc.) — updated weekly | You install exactly what you need — can include proprietary tools and licensed software |
| Scalability | Automatic — more jobs = more VMs, limited by your parallel job quota | Manual — add more machines, or use VM Scale Sets / auto-scaling runner groups for elastic scaling |
When to choose self-hosted
Self-hosted agents add maintenance complexity. Only use them when hosted agents genuinely can’t meet your requirements:
| Scenario | Why Self-Hosted | Example |
|---|---|---|
| Private network access | Pipeline needs to reach resources in a VNet, on-prem network, or behind a firewall | Deploy to a database on a private VNet, access an on-prem API |
| Specialized hardware | Build requires GPU, ARM architecture, or high-memory machines | ML model training, iOS builds on macOS, large Java builds needing 32+ GB RAM |
| Licensed software | Build tools require per-machine licenses that can’t be installed on ephemeral VMs | Licensed static analysis tools, commercial compilers |
| Long-running jobs | Jobs that exceed hosted agent time limits (6 hours for Azure, 6 hours for GitHub) | Large monorepo builds, extensive integration test suites |
| Air-gapped environments | No internet access allowed — government or defence mandate | Classified systems, SCIF environments |
| Cost optimisation at scale | Organisation runs thousands of pipeline minutes daily — self-hosted VMs are cheaper than per-minute billing | Large enterprise with 50+ concurrent builds |
Scenario: Dr. Amira's air-gapped defence pipeline
🏛️ Dr. Amira Hassan is designing a CI/CD pipeline for Major Collins’s classified logistics system. The build environment is in a SCIF (Sensitive Compartmented Information Facility) with zero internet access.
Constraints:
- No outbound internet — cannot use Microsoft-hosted agents or pull packages from public registries
- All tools must be pre-installed on the agent machines
- Agent machines must be domain-joined and meet STIG compliance
Amira’s design:
- Self-hosted agents on hardened Windows Server VMs inside the classified network
- Pre-baked VM template (golden image) with all build tools (.NET SDK, Node.js, test frameworks) pre-installed and validated
- Private Azure Artifacts feed mirrored from an internet-connected staging area — packages are scanned, approved, and transferred to the air-gapped network via a data diode
- Agent pools — separate pools for build agents (4 VMs) and deployment agents (2 VMs in the production network segment)
- Template refresh — quarterly rebuild of the golden image with updated tools, validated by Marcus (audit) before deployment
Farah (junior consultant) asks: “Why not just allow outbound HTTPS for package downloads?” Major Collins: “Because the network classification prohibits ANY outbound traffic. Period.”
Agent pools and runner groups
Azure DevOps — agent pools
Agent pools group agents for assignment to pipelines:
| Pool Type | Description | Use Case |
|---|---|---|
| Azure Pipelines (hosted) | Microsoft-managed hosted agents | Default for most pipelines |
| Default | Default pool for self-hosted agents | Simple self-hosted setups |
| Custom pools | Named pools you create (e.g., “Linux-Build”, “Windows-Deploy”) | Segmenting agents by OS, capability, network zone, or team |
Pipelines reference pools with the pool: keyword:
pool:
name: 'Linux-Build'
demands:
- docker
- Agent.OS -equals LinuxDemands allow you to match jobs to agents with specific capabilities — installed software, OS type, or custom labels.
GitHub Actions — runner groups
Runner groups are available on GitHub Enterprise:
- Group self-hosted runners by team, department, or environment
- Control which repositories can use which runner groups
- Apply organisation-level policies
Standard (non-Enterprise) accounts use labels to target specific runners:
runs-on: [self-hosted, linux, gpu]VM Scale Set agents (Azure DevOps)
For elastic self-hosted capacity, Azure DevOps integrates with Azure VM Scale Sets:
- Azure DevOps manages scaling — spins up VMs when jobs queue, shuts them down when idle
- VMs use a custom image (your golden image with pre-installed tools)
- Configure min/max agent count and idle timeout
- Combines the cost advantage of self-hosted (no per-minute charge) with the elasticity of hosted agents
GitHub Larger Runners
GitHub offers larger runners as a middle ground:
- GitHub-managed (no maintenance) but with more resources (up to 64 vCPU, 256 GB RAM)
- Available on Team and Enterprise plans
- Can be assigned static IP ranges for firewall allowlisting
- GPU runners available for ML/AI workloads
Hybrid architectures
Many organisations need both hosted and self-hosted agents in the same pipeline. A common pattern:
Stage 1: Build (Microsoft-hosted)
→ Compile code, run unit tests, publish artefacts
→ Clean environment, no special access needed
Stage 2: Integration Test (self-hosted in VNet)
→ Run tests against private database, internal APIs
→ Requires VNet connectivity
Stage 3: Deploy (self-hosted in production network)
→ Deploy to production servers behind firewall
→ Requires production network accessEach stage specifies a different pool: — the pipeline orchestration handles passing artefacts between stages.
Exam tip: agent decision framework
When the exam presents an agent selection scenario, evaluate in this order:
- Network — does the pipeline need access to private resources? If yes, self-hosted is likely required.
- Software — does the build need licensed or proprietary tools? If yes, self-hosted with pre-installed tools.
- Hardware — does the build need GPU, ARM, or high-memory? If yes, self-hosted or GitHub Larger Runners.
- Time — does the job exceed the 6-hour hosted limit? If yes, self-hosted.
- Cost — at scale (thousands of minutes/day), self-hosted VMs can be cheaper than per-minute billing.
- If none of the above apply — use Microsoft-hosted / GitHub-hosted. Simpler, cleaner, zero maintenance.
The exam strongly favours hosted agents as the default answer unless the scenario specifically requires self-hosted features.
Knowledge check
Dr. Amira's defence client requires builds to run in a network with zero internet access. All tools must be pre-installed and agents must meet STIG compliance. Which agent strategy should she recommend?
Kai's startup runs 200 pipeline minutes per month across 3 repositories on GitHub. All builds are standard Node.js projects with no special requirements. Which runner strategy minimises cost and maintenance?
Nadia's pipeline at Meridian Insurance needs to: (1) build code without special access, (2) run integration tests against a private SQL database in a VNet, and (3) deploy to on-prem servers. What agent architecture should she use?
🎬 Video coming soon
Next up: Multi-Stage Pipelines — design reusable templates, configure variables and variable groups, and implement environment-based approvals for enterprise deployment workflows.