Flock Safety's AI Search Tool for Police Can Discourage But Not Prevent Abuse, WIRED Analysis Finds
An examination of Flock's camera surveillance software reveals guardrails that record misuse but do not stop it, and delegate accuracy risk to officers despite acknowledged AI limitations.
What happened
WIRED obtained and analyzed Flock Safety's police surveillance software, which enables officers to search camera footage using AI-powered tools including a watchlist feature that conducts automated searches for people matching written descriptions across cordoned geographic areas. The company announced reforms in August—including shortened data retention defaults, required case codes for searches, and automated auditing—in response to high-profile misuse cases, including a Texas deputy's search of over 83,000 cameras for a woman who had an abortion and Illinois's discovery that federal immigration agents accessed state camera data illegally. WIRED's code analysis found that Flock's safeguards will record but not prevent abusive searches, and that the company's AI model makes consequential decisions about which person descriptions officers may search for without external oversight or measurable accuracy metrics. Experts say Flock has tasked its AI with a job no technology performs reliably at scale, and the system treats race and religion as flagged categories but exempts "political, social, and cultural expression" from automated stops—the inverse of First Amendment protections.
Context
Flock's surveillance infrastructure now sits at the center of dozens of police-stalking cases, prompting police department contract cancellations and city council votes to remove cameras. The company places the risk of inaccurate or incomplete search results on individual officers, yet the AI model that filters and ranks results operates on Flock's servers beyond departmental or public view. The gap between what Flock's guardrails can audit and what they can prevent creates a scenario where officers receive warnings about sensitive searches but can proceed anyway—a structure that shifts accountability downstream while keeping the model's logic, training data, and error rates opaque. The exemption of political and cultural expression from automated stops, while race and religion trigger system intervention, raises questions about whether technical safeguards align with constitutional protections most closely guarded by courts.
What's disputed
Flock stated its tools include "clear safeguards" and that queries are evaluated against content policies, but did not provide details on how the AI model was built, what instructions it follows, how category decisions are separated, or how long flagged-search records are retained and used internally.