Custom Detections in Defender XDR
Turn your best hunting queries into always-on detections. Learn how to create and manage custom detection rules using Advanced Hunting in Microsoft Defender XDR.
What are custom detection rules?
Think of a security guard doing rounds. Every night they walk the building checking the same things — doors locked, cameras recording, alarm panels green. They are running the same checklist on a schedule.
Custom detection rules are that scheduled checklist, but for your data. You write a KQL hunting query that finds something suspicious — say, “PowerShell downloading files from unknown domains at 3 AM.” Then you save it as a custom detection rule that runs automatically every hour, every day, or every 24 hours.
When the query finds results, Defender XDR creates an alert and an incident. No analyst needs to manually run the query — it runs itself, forever.
Creating a custom detection rule
Step 1: Write the hunting query
Start in Advanced Hunting and write a KQL query that identifies the threat:
DeviceProcessEvents
| where Timestamp > ago(1h)
| where FileName == "powershell.exe"
| where ProcessCommandLine has_any ("Invoke-WebRequest", "wget", "curl")
| where ProcessCommandLine has_any (".exe", ".dll", ".ps1", ".bat")
| where InitiatingProcessFileName != "sccm.exe" // Exclude known-good
| project Timestamp, DeviceName, AccountName, ProcessCommandLine, InitiatingProcessFileNameThis query finds PowerShell commands downloading executables — a common malware delivery technique.
Step 2: Save as detection rule
When your query works, click Create detection rule and configure:
| Setting | What to Configure |
|---|---|
| Name | Descriptive name (e.g., “PowerShell downloading executables from unknown sources”) |
| Frequency | How often the query runs: Every hour, Every 3 hours, Every 12 hours, Every 24 hours |
| Alert title | What analysts see in the alert queue |
| Severity | Informational, Low, Medium, High |
| Category | MITRE ATT&CK tactic (Execution, Initial Access, etc.) |
| Description | What the detection finds and why it matters |
| Entity mapping | Map query columns to entities: Device, User, File, IP, URL |
| Actions | What happens when the detection fires (isolate device, collect investigation package, etc.) |
Step 3: Map entities
Entity mapping is critical — it tells Defender XDR which columns in your query represent which entities. This enables:
- Automatic incident correlation (alerts sharing entities get grouped)
- Entity pages (clicking a device name shows its full timeline)
- Automated investigation on the mapped entities
Exam tip: entity mapping is required
A custom detection rule without entity mapping still creates alerts, but those alerts cannot be correlated into incidents and cannot trigger automated investigation on specific entities.
If an exam question asks “why are custom detection alerts not being grouped into incidents?” — check if entity mapping is configured.
Managing custom detections
Frequency considerations
| Frequency | Best For | Lookback |
|---|---|---|
| Every hour | Time-sensitive threats (active attacks, C2 beaconing) | Query should use ago(1h) |
| Every 3 hours | Moderate urgency (suspicious sign-ins, data exfiltration patterns) | Query should use ago(3h) |
| Every 12 hours | Low urgency, high volume (policy violations, shadow IT) | Query should use ago(12h) |
| Every 24 hours | Compliance checks, trend analysis | Query should use ago(24h) |
Detection rule actions
Custom detections can take automated response actions when they fire:
| Action | What It Does |
|---|---|
| Isolate device | Cuts network access (keeps Defender connection) |
| Collect investigation package | Gathers forensic data from the device |
| Run antivirus scan | Triggers a full Defender Antivirus scan |
| Restrict app execution | Blocks non-Microsoft-signed apps from running |
| Initiate investigation | Starts an automated investigation on the entity |
Scenario: Tyler's custom detection for CipherStack
Tyler at CipherStack writes a custom detection for a developer credential theft technique he discovered during threat hunting:
Query: Find processes reading SSH private keys from .ssh directories on developer machines
DeviceFileEvents
| where Timestamp > ago(1h)
| where FolderPath has ".ssh"
| where FileName in ("id_rsa", "id_ed25519", "config")
| where ActionType == "FileRead"
| where InitiatingProcessFileName !in ("ssh.exe", "git.exe", "code.exe")
| project Timestamp, DeviceName, AccountName, FolderPath, InitiatingProcessFileNameConfiguration:
- Frequency: Every hour
- Severity: High
- Category: Credential Access (MITRE T1552)
- Entity mapping: DeviceName → Device, AccountName → User
- Action: Collect investigation package
Tyler’s query caught a legitimate threat two weeks later — an attacker tool reading SSH keys from a compromised developer workstation.
| Feature | Custom Detections (Defender XDR) | Analytics Rules (Sentinel) |
|---|---|---|
| Engine | Defender XDR Advanced Hunting | Sentinel Log Analytics |
| Data source | Defender tables (Device, Email, Identity, CloudApp) | Any data in the Sentinel workspace |
| Query language | KQL | KQL |
| Response actions | Isolate device, collect package, run AV, restrict apps | Run playbook, assign, tag, close incident |
| Entity mapping | Device, User, File, IP, URL, Mailbox | Account, Host, IP, URL, File, and more |
| Best for | Endpoint, email, identity, and cloud app threats | Cross-source detection including third-party and custom logs |
Tyler creates a custom detection rule but notices that alerts are not being grouped into incidents with related Defender for Endpoint alerts. What is the most likely issue?
Elena at Atlas Bank wants a custom detection that runs every hour to find suspicious PowerShell activity and automatically isolates affected devices. Where does she create this?
🎬 Video coming soon
Next up: Defender XDR detections are running. Now let’s build the other half — Sentinel analytics rules that detect threats across your entire data estate.