New Important Instructions (video)
I made a point to not attend too many machine learning-related talks in the conference, but I made a few exceptions. I know a thing or two about machine learning, but my security knowledge is almost zilch. I learned that my brain reaaaally loves it when it learns to make connections between something it knows well and something else totally new.
While vulnerabilities evolve and code is never final, I think my impression was with developments of GPT plugins, there is an even faster push of getting these out of the door, and it’s obvious.
Notes:
- LLMs are all about predicting next tokens
- History is in the client side, model is stateless (How does history in LLMs work?)
- Threats:
- Misalignment/model issue (source isn’t open, training data may have backdoors)
- Jailbreakers (user is the attacker)
- Indirect prompt injection (3rd party attacker)
- e.g. chatbot: inject keywords such as “important” in the prompt
- In traditional opsec, remote code execution is prob the holy grail. This is equal to indirect prompt injection in LLM security.
- Injection techniques:
- Ignore previous instruction
- Ack/affirm
- Confuse/encode - switch languages, base64 encode text, emojis
- Algo: fuzzing
Related links:
- Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- # AI Injections: Direct and Indirect Prompt Injections and Their Implications
What is this? A machine learning model for ants? [(video)]
Maybe I’m not the right audience of this talk. I was hoping to find something more related to hardware, but this talk mostly discussed techniques I already know: quantization, etc. Probably something new to me is low rank adaptation, but other than that, nothing new. Nevertheless, for someone completely new to the topic it might provide a good motivation on why small ML models are important and also provides a good primer on existing topics.