//Question
What Should We Test in an AI Security POC for Employee AI Tools and Homegrown Agents?
Posted on 19th November, 2024

Harry
//Answer
Run one POC across both environments at once: a pilot group of employee devices and a staging copy of your homegrown AI infrastructure. Testing them separately hides the most useful result: whether one platform can cover both, or whether you’ll end up managing two inventories, two policy sets and two sets of evidence.
Test area | What the buyer should verify |
|---|---|
Employee AI discovery | Detect browser AI, desktop apps, Cursor, Claude Code, and unmanaged tools |
Homegrown agent discovery | Identify custom agents, MCP servers, models, tools, and connections |
Data protection | Detect and prevent sensitive prompts, secrets, and customer-data leakage |
Runtime enforcement | Block unauthorized tool calls before execution |
Adversarial testing | Detect prompt injection, tool misuse, and privilege escalation |
Operational impact | Measure false positives, latency, deployment effort, and developer disruption |
Governance evidence | Export inventory, violations, blocked actions, and audit records |
Two tests matter more than any dashboard walkthrough:
Blind discovery test. Before the POC, quietly install a known set of assets the vendor isn’t told about: an unapproved AI browser extension, a local stdio MCP server on a developer laptop, and an undocumented agent in staging. Then score what the platform finds on its own. Every missed asset is a gap you’d otherwise discover in production.
Blocked-action demonstration. Trigger a prohibited tool call in each environment, such as a coding agent running a destructive shell command, or a staging agent calling a tool outside its role. Pass only if the downstream log shows zero executions. An alert that fired after the call ran is detection, not prevention.
Agree pass/fail criteria before the POC starts. Run it on your own devices and data rather than a vendor demo environment, and include developers in the pilot group so disruption is measured, not assumed.
Akto covers both environments in one platform: Akto Atlas for employee devices, and Akto Argus for homegrown agents, MCP servers and GenAI apps, with one place for policies, findings and audit records.
Comments