Generative AI, Knowledge Sources & Prompt Engineering
Agents need the right knowledge and the right prompts to give accurate answers. Learn how to design knowledge sources for Copilot Studio agents, build enterprise prompt libraries, and apply prompt engineering techniques that work at scale.
The three ingredients of a great agent
A great agent needs three things: intelligence (the AI model), knowledge (the data it can access), and instructions (the prompts that guide its behaviour).
The AI model is the engine — it can understand language and generate responses. But without knowledge, it makes things up. And without good prompts, it gives vague, unhelpful answers.
As an architect, your job is to connect the right knowledge sources, design prompts that get consistent results, and build a library of reusable prompts that the whole organisation can use.
Knowledge sources in Copilot Studio
When you enable generative AI in a Copilot Studio agent, you connect it to knowledge sources — these are the data stores the agent searches before generating a response.
| Feature | What It Provides | Setup Complexity | Best For |
|---|---|---|---|
| SharePoint sites | Internal documents, policies, procedures, wikis | Low — just point to the site URL | HR agents, IT help desk, policy Q&A |
| Dataverse tables | Structured business data from D365 or custom apps | Medium — requires table selection and column mapping | Product catalogue agents, order status |
| Public websites | External web content — product pages, documentation | Low — provide URLs, agent crawls them | Customer-facing FAQ, product information |
| Custom data (files) | Uploaded documents — PDFs, Word docs | Low — drag and drop | Quick prototypes, small document sets |
| Enterprise data (connectors) | External systems — Salesforce, ServiceNow, Jira, etc. | High — Power Platform connector config and auth | Cross-system agents needing external data |
| Azure AI Search index | Vector-indexed document collections for semantic search at scale | High — requires AI Search setup and indexing pipeline | Large-scale RAG across thousands of documents |
Architecture decision: when to use which knowledge source
The exam often presents a scenario and asks which knowledge source to configure:
- “Internal company policies in SharePoint” — SharePoint sites
- “Product data in Dynamics 365” — Dataverse tables
- “Customer-facing product documentation on the website” — Public websites
- “Need to search across 50,000 documents with semantic similarity” — Azure AI Search index
- “Agent needs to pull data from Salesforce” — Enterprise data connectors
Key principle: Keep knowledge sources as close to the source of truth as possible. Don’t copy SharePoint content into uploaded files — point directly to SharePoint so the agent always has current data.
Generative AI in Copilot Studio: how it works
Copilot Studio offers two modes for handling user questions:
| Mode | How It Works | When to Use |
|---|---|---|
| Topic-based (classic) | Predefined conversation flows with branching logic. Agent follows a script | Structured processes — booking, ordering, form filling |
| Generative AI (gen AI) | Agent searches knowledge sources and generates natural language answers dynamically | Open-ended Q&A — policy questions, product inquiries |
Most production agents use both modes together: topics for structured workflows, generative AI for the “everything else” fallback. When no topic matches, the generative AI orchestrator kicks in and searches the connected knowledge sources.
Deep dive: generative AI fallback design
The exam tests your understanding of fallback behaviour in Copilot Studio:
- User sends a message
- Copilot Studio checks for a matching topic (exact or intent-based)
- If a topic matches — follow the topic flow
- If no topic matches — trigger generative AI fallback
- Generative AI searches knowledge sources — generates a grounded response
- If knowledge sources have no relevant content — return a configurable fallback message
Architect’s design decision: How much should be topic-driven vs generative?
- High-stakes processes (financial transactions, medical triage) — topics with strict flows
- Knowledge retrieval and Q&A — generative AI with good knowledge sources
- Mixed scenarios — topics for the structured parts, generative fallback for everything else
Building an enterprise prompt library
A prompt library is a governed collection of reusable prompts that teams across the organisation can use. Instead of every team writing their own prompts from scratch, the library provides tested, approved templates.
| Prompt Type | Example | Governance Level |
|---|---|---|
| System prompts for agents | ”You are a customer service agent for Apex Industries. Answer using only the provided product documentation…” | High — reviewed by AI CoE |
| Task prompts for business users | ”Summarise this meeting transcript focusing on action items and owners” | Medium — curated and tagged |
| Analysis prompts | ”Analyse this sales pipeline and identify the top 3 deals at risk of slipping” | Medium — domain-validated |
| Generation prompts | ”Write a professional email to the supplier explaining the delivery delay” | Low — self-service with guardrails |
Prompt library design guidelines:
- Version control — prompts evolve. Track versions and allow rollback
- Categorisation — organise by department, use case, and complexity
- Testing — every prompt should be tested against representative data before publishing
- Governance — system prompts require AI CoE review; task prompts can be self-service
- Metrics — track usage, satisfaction, and accuracy over time
Scenario: Jordan builds CareFirst's prompt library
Jordan Reeves creates a prompt library for CareFirst Health:
System prompts (AI CoE reviewed):
- Patient scheduling agent system prompt — includes strict guardrails about not providing medical advice
- Clinical summary generator — designed to summarise patient interactions for shift handoff
Task prompts (curated by Dr. Amara):
- “Summarise this patient interaction for the shift handoff report”
- “List all follow-up actions mentioned in this clinical note”
Guardrail: Every prompt in the CareFirst library includes a responsible AI footer: “This agent provides scheduling and administrative support only. For medical advice, please consult your healthcare provider.”
Versioning: When Jordan updates the scheduling system prompt, the old version is archived but available for rollback.
Prompt engineering for business solutions
The exam expects you to know prompt engineering at an architectural level — not just “write better prompts,” but how to design prompting strategies for enterprise AI.
| Technique | What It Does | Example |
|---|---|---|
| System message design | Sets the agent’s persona, scope, and rules | ”You are Apex’s supply chain advisor. Only answer questions about inventory and procurement.” |
| Few-shot examples | Shows the model desired input/output patterns | Include 2-3 examples of good responses in the system message |
| Chain of thought | Asks the model to reason step-by-step | ”Think through the decision step by step before providing your recommendation” |
| Output formatting | Specifies the response structure | ”Respond with a table containing: recommendation, confidence, reasoning” |
| Guardrails and constraints | Defines what the agent must NOT do | ”Never provide financial advice. If asked, redirect to the compliance team.” |
| Grounding instructions | Tells the model to only use provided context | ”Answer using only the information from the attached documents. If the answer isn’t there, say so.” |
Exam tip: prompt engineering at the architecture level
The exam isn’t about writing individual prompts — it’s about designing prompting strategies:
- Consistency: How do you ensure agents across the enterprise give consistent answers? Standard system prompts from the prompt library
- Safety: How do you prevent agents from going off-script? Guardrails, grounding instructions, content filters
- Quality: How do you improve answer quality over time? Feedback loops, A/B testing prompts, metrics
- Scalability: How do you manage prompts across 50 agents? Prompt library with versioning and governance
If a question asks “how should the architect ensure consistent AI responses across departments?” — the answer is a governed prompt library, not telling teams to “write better prompts.”
Flashcards
Knowledge check
Kai's client (Apex Industries) has 50,000 technical product specifications stored across multiple SharePoint libraries. The customer service agent needs to search these documents semantically to answer detailed technical questions. Which knowledge source should Kai configure?
Adrienne's compliance team at Vanguard wants to ensure all AI agents across the company follow consistent communication guidelines and never provide unauthorised financial advice. Which approach should Adrienne recommend?
A Copilot Studio agent for an HR department needs to handle both structured onboarding workflows (collecting documents, scheduling orientation) and open-ended policy questions from new hires. How should the architect design the agent?
🎬 Video coming soon
Next up: Small Language Models & Model Selection — when to use SLMs, how model routing works, and how to select the right model for each scenario.