Syslog, CEF & Azure Data Ingestion
Your SOC needs more than Windows events. Learn how to ingest Linux Syslog, firewall CEF logs, Azure activity data via diagnostic settings, and create custom log tables for non-standard sources.
Beyond Windows: the rest of your data
Your SOC is not Windows-only.
Linux servers send logs in a format called Syslog. Firewalls, proxies, and network appliances use CEF (Common Event Format). Azure itself generates activity logs every time someone creates a VM, changes a network rule, or modifies a policy.
And sometimes you have data that does not fit any standard format — maybe a custom application writing JSON logs, or a third-party SaaS exporting CSV. For those, you create custom log tables.
This module covers all four: Syslog, CEF, Azure activity logs, and custom tables. Together with Windows events from the previous module, these cover the majority of data sources you will configure for the exam.
Syslog via AMA
Syslog is the universal logging standard for Linux and Unix systems. Messages are categorised by facility (kern, auth, daemon, etc.) and severity (Emergency through Debug).
How Syslog ingestion works
- Install AMA on a Linux machine (the log source itself, or a dedicated forwarder)
- Create a DCR that specifies which facilities and severity levels to collect
- AMA listens on the local Syslog socket and forwards matching events to Sentinel
- Data lands in the Syslog table in your workspace
DCR filtering for Syslog
Just like Windows events, you filter Syslog at the source with a DCR:
| Facility | What It Captures |
|---|---|
| auth / authpriv | Authentication events (SSH logins, sudo) |
| kern | Kernel messages (hardware, driver issues) |
| daemon | Background services |
| local0–local7 | Custom application logs |
You select facilities and minimum severity levels. For example: collect auth at Warning and above to capture failed logins without flooding the workspace with informational messages.
CEF via AMA
Common Event Format (CEF) is a standardised log format used by firewalls, intrusion detection systems, and network appliances. CEF messages ride on top of Syslog.
CEF vs raw Syslog
| Feature | Syslog | CEF |
|---|---|---|
| Format | Free-text or structured, varies by application | Standardised key-value pairs with defined fields |
| Typical sources | Linux servers, containers, Unix apps | Firewalls (Palo Alto, Fortinet), WAFs, IDS/IPS, proxies |
| Sentinel table | Syslog | CommonSecurityLog |
| Parsing | Requires custom parsing for structured queries | Pre-parsed into named columns (DeviceVendor, Activity, etc.) |
| AMA connector | Syslog via AMA | CEF via AMA (uses Syslog transport) |
CEF ingestion architecture
Network device → sends CEF over Syslog → Linux forwarder with AMA → Sentinel (CommonSecurityLog table)Most network appliances cannot run AMA directly. Instead, you set up a Linux forwarder (a dedicated VM) that:
- Receives CEF messages on port 514 (TCP/UDP)
- AMA on the forwarder collects from the local Syslog daemon
- Forwards to the Sentinel workspace
Scenario: Anika's firewall ingestion
Three of Anika’s MSSP clients use Palo Alto firewalls. The firewalls send CEF-formatted threat logs over Syslog.
Anika deploys one Linux forwarder VM per client (for data isolation) with AMA installed. Each forwarder has a DCR that filters for the relevant CEF severity levels. The firewall logs land in the CommonSecurityLog table, where Anika’s analytics rules detect port scanning, command-and-control traffic, and data exfiltration attempts.
Azure activity logs
Azure Activity Logs record every control-plane operation in your Azure subscription: who created a VM, who changed a network security group, who deleted a storage account.
Two ways to collect Azure activities
| Method | How It Works | Best For |
|---|---|---|
| Azure Policy | Assigns a diagnostic setting to all resources in a subscription automatically | Large environments, consistent deployment |
| Resource diagnostic settings | Configure per-resource or per-subscription in Azure Monitor | Targeted collection from specific resources |
Azure Policy is the recommended approach for the exam — it ensures every new resource automatically sends its diagnostic data to Sentinel without manual configuration.
What Azure activity logs capture
- Administrative — resource create, update, delete operations
- Security — Defender for Cloud alerts, security recommendations
- Service Health — Azure service issues affecting your resources
- Alert — Azure Monitor alert activations
- Policy — Azure Policy evaluations and compliance changes
Exam tip: Azure Policy vs manual diagnostic settings
If the exam asks “how to ensure ALL Azure resources send diagnostic data to Sentinel,” the answer is Azure Policy with a built-in policy definition like “Deploy diagnostic settings for Activity Log to Log Analytics workspace.”
If the question asks about a specific resource (e.g., “a Key Vault”), the answer is resource diagnostic settings configured on that resource.
Custom log tables
When your data does not fit any built-in connector, you create a custom log table using the Logs Ingestion API.
When to use custom log tables
- Third-party SaaS that exports logs via API but has no Sentinel connector
- Custom applications writing JSON or structured logs
- Legacy systems with proprietary log formats
- Enrichment data (threat intel feeds, asset inventories) you want queryable in KQL
How custom log ingestion works
- Create a custom table in the Log Analytics workspace (table name must end with
_CL) - Create a DCR with a transformation that maps your data to the table schema
- Send data to the Logs Ingestion API endpoint using the DCR ID
- Data is transformed and stored in the custom table
The DCR transformation uses KQL-based transformations to reshape incoming data — rename fields, parse timestamps, filter rows.
Scenario: Elena's ATM transaction logs
Elena at Atlas Bank needs ATM transaction logs in Sentinel for fraud detection. The ATM management system exports JSON logs via an API but has no Sentinel connector.
Elena creates:
- A custom table: ATMTransactions_CL with columns for transaction_id, card_type, amount, location, timestamp
- A DCR with a KQL transformation that parses the JSON and maps fields
- A Logic App that polls the ATM API every 5 minutes and sends data to the Logs Ingestion API
Now she can write KQL queries and analytics rules against ATM data alongside security events.
Anika needs to ingest Palo Alto firewall threat logs into Sentinel. The firewalls send CEF-formatted messages over Syslog. Which architecture should she use?
James wants to ensure that every Azure resource in Pacific Meridian's subscription sends diagnostic data to Sentinel — including resources created in the future. What should he configure?
Anika manages 40 MSSP clients. A new client wants CEF firewall logs AND Azure activity logs in Sentinel. The client also has a custom IoT platform that writes JSON logs via API. Anika must minimise cost and complexity. Which combination of connectors should she use?
🎬 Video coming soon
Next up: Data is flowing into Sentinel. Now let’s configure the other half of the equation — Microsoft Defender for Endpoint — and make sure it’s set up for advanced detection and response.