Benchmark your AI security readiness for 2027 → [Take the survey]

Benchmark your AI security readiness for 2027 → [Take the survey]

Benchmark your AI security readiness for 2027 → [Take the survey]

//Question

What Should We Test in an AI Security POC for Employee AI Tools and Homegrown Agents?

Posted on 19th November, 2024

Harry

Harry

//Answer

Run one POC across both environments at once: a pilot group of employee devices and a staging copy of your homegrown AI infrastructure. Testing them separately hides the most useful result: whether one platform can cover both, or whether you’ll end up managing two inventories, two policy sets and two sets of evidence.

Test area

What the buyer should verify

Employee AI discovery

Detect browser AI, desktop apps, Cursor, Claude Code, and unmanaged tools

Homegrown agent discovery

Identify custom agents, MCP servers, models, tools, and connections

Data protection

Detect and prevent sensitive prompts, secrets, and customer-data leakage

Runtime enforcement

Block unauthorized tool calls before execution

Adversarial testing

Detect prompt injection, tool misuse, and privilege escalation

Operational impact

Measure false positives, latency, deployment effort, and developer disruption

Governance evidence

Export inventory, violations, blocked actions, and audit records

Two tests matter more than any dashboard walkthrough:

Blind discovery test. Before the POC, quietly install a known set of assets the vendor isn’t told about: an unapproved AI browser extension, a local stdio MCP server on a developer laptop, and an undocumented agent in staging. Then score what the platform finds on its own. Every missed asset is a gap you’d otherwise discover in production.

Blocked-action demonstration. Trigger a prohibited tool call in each environment, such as a coding agent running a destructive shell command, or a staging agent calling a tool outside its role. Pass only if the downstream log shows zero executions. An alert that fired after the call ran is detection, not prevention.

Agree pass/fail criteria before the POC starts. Run it on your own devices and data rather than a vendor demo environment, and include developers in the pilot group so disruption is measured, not assumed.

Akto covers both environments in one platform: Akto Atlas for employee devices, and Akto Argus for homegrown agents, MCP servers and GenAI apps, with one place for policies, findings and audit records.

Comments