Agent Tools & Knowledge Integration
Agents are only as useful as the tools they can access. Learn how to integrate APIs, knowledge stores, search, Content Understanding, and custom functions into your AI agents.
Giving agents superpowers
An agent without tools is like a brilliant person locked in an empty room — smart but useless. Tools let agents interact with the real world.
Tools are the bridges between your agent’s reasoning and your actual systems. Need to search a database? Tool. Extract data from a PDF? Tool. Send an email? Tool. Call an external API? Tool. Each tool is a well-defined function the agent can decide to call.
The five tool categories
| Category | What It Does | When to Use | Example |
|---|---|---|---|
| APIs | Calls external services via REST | Real-time data, third-party services | CRM lookup, payment processing, weather API |
| Knowledge stores | Searches indexed documents | Q&A over company docs | Foundry IQ with policy documents |
| Search | Performs sophisticated queries | Complex retrieval with ranking | Azure AI Search with hybrid search |
| Content Understanding | Extracts data from documents | Processing uploaded files | Extract fields from invoices, contracts |
| Custom functions | Runs your own code | Business logic, calculations, data transforms | Calculate shipping cost, validate input, format output |
Built-in vs custom tools
| Feature | Built-in Foundry Tools | Custom Functions |
|---|---|---|
| Examples | Code interpreter, web search (Bing), Foundry IQ | Your own APIs, database queries, business logic |
| Setup | Enable in agent configuration | Write function code, define schema, register with agent |
| Hosting | Managed by Foundry | Your infrastructure (Container Apps, Functions, etc.) |
| Best for | Common capabilities out of the box | Domain-specific actions unique to your business |
Content Understanding as an agent tool
Content Understanding is particularly powerful as an agent tool because it lets agents process documents in real-time:
| Capability | Agent Use Case |
|---|---|
| OCR | Agent reads a photo of a receipt or form uploaded by the user |
| Layout analysis | Agent understands table structures in uploaded spreadsheets |
| Field extraction | Agent pulls specific values (invoice number, total, date) |
| Document classification | Agent identifies what type of document was uploaded |
Real-world example: NeuralMed's intake agent
NeuralMed’s patient intake agent uses multiple tool types:
- Content Understanding — patient uploads insurance card photo → agent extracts member ID, group number, provider
- Custom function —
verify_insurance(member_id, provider)→ checks if insurance is active - Knowledge store — searches NeuralMed’s service catalog to match patient needs with available services
- API —
check_availability(doctor_id, date_range)→ finds available appointment slots - Custom function —
book_appointment(patient_id, doctor_id, slot)→ books the appointment
Five tools, one seamless patient experience. The agent orchestrates the entire flow.
Tool design best practices
| Practice | Why It Matters |
|---|---|
| Descriptive names | Agent understands when to use each tool |
| Clear parameter docs | Agent passes correct arguments |
| Error handling | Agent can recover gracefully when a tool fails |
| Minimal scope | Each tool does one thing well (single responsibility) |
| Idempotent writes | Calling the same tool twice doesn’t create duplicates |
Exam tip: Content Understanding vs prompting a multimodal model
The exam tests when to use Content Understanding vs a multimodal model for document processing:
- Extract specific fields from structured documents (invoices, forms) → Content Understanding (purpose-built, higher accuracy)
- Understand and reason about document content → Multimodal model (GPT-4o with vision)
- Both needed? → Content Understanding extracts data, then the model reasons about it
Rule: extraction = Content Understanding, reasoning = multimodal model.
Key terms
Knowledge check
MediaForge's content agent receives a client brief as a PDF. The agent needs to extract the target audience, budget, and timeline fields, then generate 5 headline ideas based on the brief. Which tools should the agent use?
Atlas Financial's compliance agent needs to verify a customer's identity by checking their driver's license. The customer uploads a photo. Which tool combination is appropriate?
🎬 Video coming soon