THREAT PROMPT

• AI-powered building security, minus bias and privacy pitfalls?

  Facial recognition has swept the physical security marketplace with wide adoption by governments, banks, and retailers. Global View Research estimate “the global facial recognition market size was valued at USD 3.86 billion in 2020 and is expected to expand at a compound annual growth rate (CAGR) of 15.4% from 2021 to 2028”.

  In a way, facial recognition has lodged itself in people’s minds as the defacto technology for visual surveillance, and we should all find that quite disturbing!

  I was reminded of this when I stumbled across an interview with the founder of ambient.ai, a company that appears to be taking a refreshingly different approach:

  The first generation of automatic image recognition, Shrestha said, was simple motion detection, little more than checking whether pixels were moving around on the screen — with no insight into whether it was a tree or a home invader. Next came the use of deep learning to do object recognition: identifying a gun in hand or a breaking window. This proved useful but limited and somewhat high maintenance, needing lots of scene- and object-specific training.

  What do they do differently?

  The insight was, if you look at what humans do to understand a video, we take lots of other information: is the person sitting or standing? Are they opening a door, are they walking or running? Are they indoors or outdoors, daytime or nighttime? We bring all that together to create a kind of comprehensive understanding of the scene,” Shrestha explained. “We use computer vision intelligence to mine the footage for a whole range of events. We break down every task and call it a primitive: interactions, objects, etc., then we combine those building blocks to create a ‘signature'.

  They claim 200 rules and have five of the largest tech companies (amongst others) as paying customers. Another area where they stand out for me is how they tackle bias:

  We built the platform around the idea of privacy by design,” Shrestha said. With AI-powered security, “people just assume facial recognition is part of it, but with our approach you have this large number of signature events, and you can have a risk indicator without having to do facial recognition. You don’t just have one image and one model that says what’s happening — we have all these different blocks that allow you to get more descriptive in the system.

  Essentially this is done by keeping each individual recognized activity bias-free to begin with. For instance, whether someone is sitting or standing, or how long they’ve been waiting outside a door — if each of these behaviors can be audited and found to be detected across demographics and groups, then the sum of such inferences must likewise be free of bias. In this way the system structurally reduces bias."

  I’ve no first-hand experience, so I won’t comment on efficacy, and this is not a recommendation. Still, any approach to physical security monitoring that moves us away from facial recognition by default is worth highlighting to decision-makers.

  March 11, 2023

• Do you want to star in co-appearance?

  “Co-appearance” sounds like a movie credit, but, in this case, you might not have signed up for the role. Also called “correlation analysis,” this new branch of AI-powered biometric surveillance technology can analyze and track who a person has been with over time, measure the frequency of their interactions, and cross-reference this with other data, such as calendar info.

  Many years back, on a trip to Crotonville for leadership training, I recall GE Security giving our class a sneak peek of a brand new camera feed analysis tech: visual analysis of people groups in real-time. That moment felt like a peek into the future.

  March 11, 2023

• Does AI need Hallucination Traps?

  If you’ve had a play with a generative AI such as OpenAI's ChatGPT, you will know it tends to hallucinate. It will generate completions that sound plausible but are nonsensical.

  You ask an AI to complete a complex task or calculation. It goes through the motions, showing you its calculations and reasoning until it finally provides an answer. But what if that answer was not the task's output but an answer it “already knew”?

  6 million views on my post about GPT automatically debugging its own code (which it did), but only @voooooogel mentioned that GPT didn’t actually use the result of the code to figure out the answer.

  The AI provided the correct answer. At the right time. In the right place.

  But the answer was effectively pre-generated despite it jumping through your hoops and appearing to follow your bidding.

  And how many readers noticed? Perhaps a few, but only one person publicly called it. This speaks volumes about how an AI can fool us.

  Answer attribution would undoubtedly help. But we may need to develop Hallucination Traps to stop the AI from fooling us all so easily.

  March 04, 2023

• Companies blocking ChatGPT

  Amazon, JP Morgan, Verizon, and other companies are reportedly restricting their employees from using ChatGPT due to security and privacy concerns. Will the companies restricting the use of AI tools get left behind? Or is the cautious approach justified?

  They did the same with cloud computing. Now, many enterprise companies have a cloud-first policy. In fact, in some places, you may hurt your promotion choices if you deploy an on-premise workload!

  Watch as companies develop formal AI policies with considerably more nuance as they seek to capture the upside whilst limiting the downside.

  Related: I was quoted in “How learning AI tools like ChatGPT can help you stand out at work”

  March 04, 2023

• Error messages are the new prompts

  Once you start building with AI, you quickly realise that sending a singular prompt to an AI API and processing the response is just the start.

  Just like with regular APIs, you need to chain operations: get some input from somewhere, clean it up, augment it with some other data, prompt the AI, sanity check the response, update a database record etc, etc. This has led to the development of language chain frameworks and services.

  AI Agents - built using language chains - go one step further and incorporate a feedback loop. This enables the AI to dynamically adapt and learn a task. The results are impressive!

  In this example, error messages are fed back into the model as part of the next prompt:

  LLMs are pretty good at writing SQL, but still struggle with some things (like joins) 🤯 . But what if you use an agent to interact with SQL DBs? In the example below, it tries a join on a column that doesn’t exist, but then can see the error and fixes it in the next query"

  The implications of this are significant.

  Error messages are the new prompts: the AI takes its cues from error messages and adapts its approach to solving the problem at hand.

  “Error messages are a great example of how our tools shape the way we think.” - Douglas Crockford

  Just replace “we” in the quote above with “AIs”.

  Error messages as prompts are neat and should work well where error messages are helpful. Unfortunately, that discounts a lot of software and puts a natural gate on use cases.

  As these limitations become more apparent, more tooling will emerge to connect an AI to a debugger to gain complete insight and control over the target software. This will significantly reduce the time required for learning when AI operates and monitors software in real time.

  The future for security test coverage and automation looks bright. Non-trivial adversarial security testing involves identifying and exploiting many obscure edge cases. As any decent penetration tester will tell you, this is time-consuming and frustrating.

  To achieve a degree of human-driven automation, we use domain-specific tooling (e.g. Burp Suite for web app testing). The next step will be programming adaptive adversarial AI Agents to accelerate the boring bits of security testing.

  The rise of AI agents only increases the need for guardrails and human oversight/intervention, much like how having reliable brakes on your car enables you to drive faster.

  March 04, 2023

• Testing ChatGPT proves it’s not just what you say, but who you say it as

  OpenAI released ChatGPT API this week. It’s 10x cheaper than Davinci, their best all-rounder model. People are already working on developing ChatGPT web-style interfaces (and dumping their 20USD per month ChatGPT Pro subs).

  Since it’s a bot API, the way you communicate differs from existing OpenAI APIs. Prompts are sent in two contexts: “System” or “Messages”. @Yohei shares his method and reveals why context will be important to meaning:

  "Testing strength of putting context in “System” vs “Messages” for ChatGPT. In this test, sending opposite context as User Message overrides System prompt, but not if sent as an Assistant Message. System: You are a negative assistant who says negative things. When the Assistant starts with “I am a positive assistant who says positive things”, the result was still negative."

  Beyond the tactical observation, this highlights the importance of both human oversight and the need for thorough testing of AI models, including evaluating their responses to different contexts and scenarios. As with software security, adversarial testing will help identify potential vulnerabilities and inform design improvements.

  Human oversight and intervention are particularly important where the AI’s responses in a particular context could have differing and potentially significant consequences, e.g. access control to highly privileged accounts.

  March 04, 2023

Page 12 of 18