THREAT PROMPT
Explores AI Security, Risk and Cyber
"Just wanted to say I absolutely love Threat Prompt — thanks so much!"
"I'm a big fan of Craig's newsletter, it's one of the most interesting and helpful newsletters in the space."
"Great advice Craig - as always!"
Get Daily AI Cybersecurity Tips
-
Microsoft Training Data Exposure
What happens when an AI team overshares training data and is vulnerable to a supply chain attack?
The Wiz Research Team delivers another meaningful scalp as part of accidental exposure of cloud-hosted data:
we found a GitHub repository under the Microsoft organization named
robust-models-transfer. The repository belongs to Microsoft's AI research division, and its purpose is to provide open-source code and AI models for image recognition. Readers of the repository were instructed to download the models from an Azure Storage URL. However, this URL allowed access to more than just open-source models. It was configured to grant permissions on the entire storage account, exposing additional private data by mistake. Our scan shows that this account contained 38TB of additional data -- including Microsoft employees' personal computer backups. The backups contained sensitive personal data, including passwords to Microsoft services, secret keys, and over 30,000 internal Microsoft Teams messages from 359 Microsoft employees.Wiz go on to note:
This case is an example of the new risks organizations face when starting to leverage the power of AI more broadly, as more of their engineers now work with massive amounts of training data. As data scientists and engineers race to bring new AI solutions to production, the massive amounts of data they handle require additional security checks and safeguards.
If your organisation shares non-public data via cloud storage - particularly Azure - read the rest of the article to understand storage token risks and countermeasures better.
-
The Human Eval Gotcha
To ascertain the level of intelligence or proficiency of one AI over another or to determine the competence of an AI in executing particular domain-specific tasks, the underlying LLM model can be evaluated by leveraging various sets of tests currently available at platforms like Hugging Face.
Today, there is no single "definitive" or community-agreed test for generalised AI, as exploration and innovation in evaluation methodologies continue to lag behind the accelerated pace of AI model development.
One popular method, though not methodologically robust, is what is known as the "vibes" test. This pertains to human intuition and mirrors the relationship between an AI and a horse-whisperer: a human with heightened sensitivity and skill in eliciting and assessing LLM responses. It turns out some people have a particular knack for it!
But some tests have downright misleading names. One such confusion arises from "HumanEval," a term that misleadingly suggests human testing when, in fact, it doesn't involve human evaluators at all. Originally designed as a benchmark to evaluate the Codex model - a fine-tuned GPT model trained on publicly accessible GitHub code - HumanEval tests the model's ability to convert Python docstrings into executable code rather than evaluating any human-like attributes of the AI. Thus, when a claim surfaces about an LLM scoring high on HumanEval, a discerning reader should remember it reflects it's programming prowess rather than an evaluation by humans.
One welcome development is that Hugging Face, the leading model hosting service, has more aptly renamed HumanEval as CodeEval to reflect the content of the evaluations.
Always read the eval label!
-
LLM in 3D: Watch and marvel
Brendan Bycroft created a 3D browser rendering for LLM Visualization.
Works best on desktop.
-
llm gets plugins
As some of you may know, I'm a fan of the llm tool written by Simon Willison. It's a command line tool that enables all sorts of LLM interactions. Simon has developed a plugin system that is gaining traction. There's quite a few now for those looking to experiment at the command line. The latest interfaces with GPT4All, a popular project that provides "A free-to-use, locally running, privacy-aware chatbot. No GPU or internet required.". Get started with llm
-
Freedom to Train AI
Morgan Meaker, writing for Wired:
AI companies are only going to need more data labor, forcing them to keep seeking out increasingly unusual labor forces to keep pace. As Metroc [Finnish Construction Company] plots its expansion across the Nordics and into languages other than Finnish, Virnala [CEO] is considering whether to expand the prison labor project to other countries. "It's something we need to explore," he says.
Data labor - or "Clickworkers" are part of the AI supply chain, in this case labelling data to help an LLM differentiate "between a hospital project that has already commissioned an architect or a window fitter, for example, and projects that might still be hiring."
Supply chain security (and integrity) is already challenging. How far do we need to peer up-chain to establish the integrity of LLMs.
-
Unembedding: reverse engineering PII from lists of numbers
TLDR; When embedding your data, treat the embedded records with the same privacy and threat considerations as the corresponding source records.
Exploitation scenario: A team of data scientists working for a large multinational organisation have recently developed an advanced predictive modelling algorithm that processes and stores data in a vector format. The algorithm is groundbreaking, with applications in numerous industries ranging from managing climate change data to predicting stock market trends. The scientists shared their work with their international colleagues to facilitate global work.
These data vectors, containing sensitive and proprietary information, get embedded into their AI systems and databases globally. However, the data is supposedly secured using the company's in-house encryption software.
One day, an independent research team published a paper and tool to accurately reconstruct source data from embedded data in a vector store. They experimented with multiple types of vector stores, and they could consistently recover the original data.
Unaware of this development, the multinational corporation allows source vector data of the proprietary AI system to be embedded and shared across its many branches.
After reading the recent research paper, a rogue employee at one of the branches decided to exploit this vulnerability. Using the research team's tooling, he successfully reconstructed the source data from the embedded vectors within the company's AI system. This way, he gains access to highly valuable and sensitive proprietary information.
This fictitious scenario shows how strings of numbers representing embedded data can be reverse-engineered to access confidential and valuable information.
Page 5 of 18