Editorial illustration for Narrower AI assistant, one filter close retrieval gap in UKASI test
AI Assistant Closes Retrieval Gap in UKASI Test
Egiziago Cioffi built the assistant himself. As IT and Enterprise Architect and CEO of SynSphere Italia, a Microsoft partner in Milan, he wrote the indexing job, configured the Azure OpenAI retrieval pipeline, and wired it into SharePoint. The evaluation scores came back clean.
Unit tests passed. The email assistant now auto-resolves roughly 60% of inbound customer email, Cioffi told VentureBeat in written responses to interview questions.
None of that testing asked the one question that mattered: would the assistant hand a low-privilege user documents that user could never open directly in SharePoint. Cioffi ran that exact test, comparing outputs from a low-privilege account against a high-privilege account asked the same questions. The answers didn't match. The assistant was retrieving content under the indexer's permissions, not the requester's, a failure mode that doesn't show up in accuracy metrics or automated evals.
Cioffi's logs document one production case. But the underlying gap sits inside the plumbing of Azure AI Search itself, where document-level access control depends on Entra ID tokens, SharePoint ACL sync previews, and API behavior that doesn't yet cover every principal type the same way.
Cioffi ran a low-privilege account against the same questions a high-privilege account had already put to the assistant. The outputs did not match. The assistant returned SharePoint content the requesting user could not have opened in SharePoint on their own.
Why this matters
Cioffi's fix is a useful corrective for anyone building agent pipelines right now: the gap wasn't solved by a new identity layer or a fancier retrieval stack, it was solved by narrowing what the assistant was allowed to touch and adding one filter. That's a cheap, boring fix, and boring fixes tend to get skipped in favor of more architecture. The UKASI framing matters too.
Calling it a containment failure rather than a retrieval-entitlement failure is a real distinction, and it should worry teams who assume their evaluation environment is catching scope violations. A permissive test environment with no mechanism to flag deviation will pass clean eval scores right up until the agent does something outside its intended lane in production. For developers and founders shipping Azure OpenAI or similar assistants, the takeaway isn't "add more infrastructure." It's audit scope first, test containment specifically, and don't mistake a passing eval suite for proof the agent stays inside its lines.
Related Reading
Business & Startups Nordic pilot adds Gemini for Education, NotebookLM to boost AI literacy
Business & Startups Kling launches Video O1, all-in-one model with MVL bridge using transformer
Business & Startups DeepSeek Seeks More Capital Weeks After USD 7B Funding Round
From Archives AIR Raises USD 50M to Vet AI Agent Skills, Citing Unchecked 'Same Risk
From Archives Perplexity says it isn't using confidential data for post-training AI
Common Questions Answered
What security vulnerability did Egiziago Cioffi discover in his Azure OpenAI email assistant during testing?
Cioffi discovered that the assistant was returning SharePoint content to low-privilege users that they could not have accessed on their own, revealing a containment failure in the retrieval pipeline. This critical gap went undetected through standard unit tests and evaluation scores, only becoming apparent when testing with different privilege levels against the same queries.
How did Cioffi resolve the retrieval-entitlement gap in the Azure OpenAI assistant?
Rather than implementing a new identity layer or complex retrieval architecture, Cioffi fixed the issue by narrowing what the assistant was allowed to access and adding a single filter to enforce permissions. This simple, straightforward solution proved effective without requiring extensive architectural changes to the system.
What is the significance of framing this issue as a containment failure versus a retrieval-entitlement failure?
The distinction matters because it changes how developers approach the problem and what solutions they prioritize. Calling it a containment failure emphasizes the need for access control boundaries rather than focusing solely on improving retrieval mechanisms, which can lead to more appropriate and efficient fixes.
What was the performance level of Cioffi's email assistant after implementing the security fix?
The email assistant was able to auto-resolve roughly 60% of inbound customer email after the containment fix was applied. This performance level was achieved while maintaining proper security controls to prevent unauthorized access to SharePoint content.
Further Reading
- Up to 40% better relevance for complex queries with new agentic retrieval engine - Microsoft Tech Community
- Persistent Limitations in Azure OpenAI for Multi-Document ... - Microsoft Learn Answers
- Set the Retrieval Reasoning Effort - Azure AI Search - Microsoft Learn
- Building KG-RAG Enhanced AI Assistants - arXiv
- Building an operations assistant on Azure that waits for approval before it acts - Microsoft DevBlogs
