Archive

Discussion of section 2.1 of the International AI Safety Report (January 2025) on risks from malicious use. Part of the AIGS Canada network.

Orpheus presents the core ideas from "Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems" (arxiv.org/abs/2405.06624), followed by Q&A and open discussion.

Coworking session for anyone working on AI safety — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Samuel Gélineau presents Parity Bot (gelisam.com/parity-bot): demonstrating that a neural network can be formally verified to satisfy a safety property on all inputs, by adapting a range analysis algorithm from classical code verification to neural network weights.

Coworking session for anyone working on AI safety — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Evan Lombardi discusses how social media algorithms harm mental health and how AI amplifies these effects. Covers recommendation systems, dark patterns, deepfakes, and the mis/disinformation ecosystem, grounded in the Cambridge study "Algorithms, Addiction, and Adolescent Mental Health". Closes with an introduction to the Unplug Project.

Discussion of the Global Call for AI Red Lines (red-lines.ai), launched at UNGA-80 by Nobel laureates, former heads of state, and AI pioneers including Yoshua Bengio. The initiative urges governments to negotiate by end of 2026 an international agreement prohibiting the most dangerous AI uses and behaviors.

Olivier Coutu presents aisafety.info: an introduction to existential risks from AI, a 300+ article FAQ, a friendly chatbot named Stampy, and a dataset of AI alignment resources. Initiated by Rob Miles and sustained by volunteers.

Nik Lacombe introduces the PauseAI movement and its Montréal group (pauseai.ca/montreal). PauseAI aims to mitigate AI risks by convincing governments to pause the development of superhuman AI.

What should "AGI" mean and how could we measure it? Presentation of agidefinition.ai, which operationalizes AGI as matching the cognitive versatility of a well-educated adult using the Cattell-Horn-Carroll (CHC) model — decomposing intelligence into ten abilities with concrete benchmark tasks.

Coworking session for anyone working on AI safety — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Launch event for If Anyone Builds It, Everyone Dies by Yudkowsky and Soares (ifanyonebuildsit.com) — a case for why current paths toward superhuman AI could end in catastrophe. Opens with a primer on the book's arguments, followed by discussion and Q&A on technical, policy, and institutional risk-reduction.

Coworking session for anyone working on AI safety — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Walkthrough of the First Key Update of the International AI Safety Report (Oct 14, 2025): rapid gains in maths, coding, and scientific assistance from post-training reasoning techniques; the emergence of longer-horizon tool-using agents; and implications for bio/cyber risks, controllability, and labour markets.

Coworking session for anyone working on AI safety — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Coworking session for anyone working on AI safety — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Reading group discussion of If Anyone Builds It, Everyone Dies by Yudkowsky and Soares, hosted by Jeremy. Guided by questions suggested by the authors on the three core claims: that superhuman AI is buildable, that the default outcome is extinction, and that it can still be prevented.

How does Canada's 2025 federal budget address AI risk? Discussion mapping the AI-related budget items to threat models — power concentration, epistemics, bio, autonomy, misuse, systemic risk, and more.

Coworking session for anyone working on AI safety or governance — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Workshop to co-design a National Citizens' Assembly on Superintelligence for Canada. Participants align on mandate, stakeholders, and public input mechanisms, producing draft outputs: a Concept Note, a Consortium Intent Memo, and an invite list.

Introduction to Neuronpedia (neuronpedia.org): sparse autoencoders, features, and feature pages. Live demo exploring a model slice — searching features, running activation tests, and discussing standards of evidence. Closes with paths to contribution.

48-hour hackathon on defensive acceleration (def/acc): building technology to protect against pandemics, cybercrime, and powerful AI. Organized with Apart Research. $20,000 USD in prizes. Registration through Apart Research's event page required.

Coworking session for anyone working on AI safety or governance — research, engineering, policy, or field-building. Light collaboration or quiet focus.

Activity led by Emma Kondrup. Examining pessimistsarchive.org — newspaper headlines across technological eras (automobile, radio, TV) — to interrogate AI exceptionalism: are AI's differential traits truly consequential enough to justify unique governance responses, and what would sufficient differences look like for future AGI/ASI?

Discussion of Scheffer et al.'s 2009 Nature review on critical transitions in complex systems. Despite mechanistic differences, systems approaching tipping points share generic early-warning signals — rising autocorrelation, increased variance — rooted in "critical slowing down". Facilitated by Orpheus, with implications for AI governance.

Presentation by Taylor Lynn Curtis, Mila researcher on misinformation. Explores AI systems designed for persuasion, then introduces Veracity — a real-world tool using AI to ensure data quality and detect misinformation. Closes with implications for responsible AI governance.

Presentation by Joaquim Streicher, Ph.D. candidate in Neuroscience and co-founder of MONIC (Montreal Initiative for Consciousness science). Addresses AI consciousness: current expert consensus, the challenge of distinguishing true consciousness from imitation, risks of false positives and negatives, and lessons from consciousness testing in unresponsive patients.

48-hour hackathon on AI manipulation defense — building benchmarks, detection systems, and mitigations for sycophancy, deception, sandbagging, reward hacking, and dark patterns. Organized with Apart Research. $2,000 USD in prizes + fast-track to the Apart Fellowship and presentation at IASEAI'26 in Paris.

Screening of Writing Doom, a ~30-minute short film on AI, followed by discussion. Hosted by PauseAI Montréal.

Presentation by Ariane Ollier-Malaterre (Canada Research Chair in Digital Regulation at Work, ESG-UQAM). Based on 58 qualitative interviews, her book explores how Chinese citizens make sense of and live with pervasive digital surveillance — their imaginaries, coping strategies, and the psychological weight of constant surveillance exposure.

Weekend hackathon on technical AI governance infrastructure: hardware verification, attestation systems, privacy-preserving compliance proofs, risk threshold frameworks, and international coordination mechanisms. Organized with Apart Research. $2,000 USD in prizes + fast-tracks to Lucid Computing, MIRI TGT, and Apart Fellowship.

Presentation by Rashid Mushkani (UdeM + Mila). Discussion on who decides model behavior and by what criteria.

Coworking session for anyone working on AI safety or governance, followed by bouldering at Bloc Shop Mile-Ex. Entry ~$13 on Fridays. Bring your own lunch.

Reading circle of the OBVIA Cybersecurity & Cyberjustice axis. Presentation by Félix Tréguer, response by Benoît Dupont.

Participatory workshop (game format) on five issues of generative AI: information reliability, personal data, intellectual property, algorithmic bias, environmental impact. Registration required.

Workshop on thoughtful and ethical AI use, part of the free Perspective AI student program.

Meeting of the AI safety community of practice.

Workshop on data sharing and security in the AI era, part of the free Perspective AI student program.

Presentation by Benoît Dupont (Canada Research Chair in Cyber-Resilience, UdeM). Analysis of 160+ cybercriminal forum conversations: how hackers perceive and exploit AI, their doubts, and the implications for security.

Bring your laptop and your climbing shoes. Work on AI safety projects in good company, then boulder together.

Panel discussion on rights balancing: how frameworks protecting AI workers also reinforce human rights protections. Speakers include Heather Alexander and Jonathan Simon.

Community roundtable for the People's Consultation on AI. Share and discuss concerns about the impact of robotics and AI across sectors and what we want the government to do. No technical background needed. Limited FR/EN translation available.

Hands-on workshop to build a neural network in 2 hours, part of the free Perspective AI student program.

Shalaleh Rismani (McGill/Mila) presents research on how humans actually oversee AI systems — finding that greater understanding doesn't necessarily lead to better oversight.

Working on something related to AI safety or governance (e.g. research, engineering, policy)? Come work on your projects alongside others in the field — and boulder between sessions. We meet at Blocshop Mile Ex, 10h-14h. Entry ~$13 on Fridays. The gym has a café with tables & wifi. You may want to bring a lunch.

Workshop on reasoning in multi-agent systems: theory of mind, argumentation, and distributed reasoning — 22 speakers from AI, neuroscience, and philosophy. Part of IVADO's thematic semester on computational reasoning. $40–$240.

Overview of the Montréal AI safety landscape — who's doing what, where the bridges and gaps are — followed by structured small-group discussions on what's missing, who should be talking, and what to do in the next few months.

Week-long hackathon with three tracks: adversarial stress-testing, logic hardening, and synthetic data augmentation for safer conversational AI. Prizes include $10K and a Mila AI Safety Studio internship.

Working session to critically review CAIM's initial records — debate borderline cases, calibrate severity ratings, identify gaps, and flag records needing further work.

A discussion on artificial intelligence and the notion of consciousness — where does it begin?

Three-day hackathon to develop control protocols, build evaluation tools, and stress-test safety measures using ControlArena and SHADE-Arena.

Max (Greyhaven) presents Greywall, a software-defined sandbox and proxy for AI agents that provides fine-grained runtime visibility and controls. The talk also explores how sandboxing connects to the field of AI control.

Colloquium on integrating AI and sovereign digital technologies into government while protecting ethics, transparency, and public trust.

Jonathan Durand Folco (University of Saint-Paul) and Jonathan Martineau (Concordia) discuss algorithmic capitalism, its concentration of power, and possible forms of resistance.

Jean-François Godbout (Université de Montréal + Mila) explores the persuasive effects of generative AI, including deepfakes, automated influence, and AI-assisted propaganda, alongside mitigation strategies and a new AI-powered misinformation detection tool.

International colloquium on the ethical, legal, and social issues raised by generative AI in sexuality and intimate relationships.

Five-day bootcamp on statistical foundations of modern AI, part of the IVADO thematic semester on Statistical Foundations of AI.

Five-day workshop on statistical methods for trustworthy AI, part of the IVADO thematic semester on Statistical Foundations of AI.

The Montréal node of the Secure Program Synthesis Hackathon, a weekend research sprint at Ω Labs, organized with Apart Research and Atlas Computing. Build a prototype at the intersection of program synthesis, formal methods, security. Tracks - Specification Elicitation: tools that pull formal specs out of ambiguous sources (docs, legacy code, requirements). - Specification Validation: methods that check a candidate spec actually captures intended behavior. - Spec-Driven Development / Vericoding: workflows where a spec generates and ranks candidate implementations. - Adversarial Robustness for ITPs and proof tools: fuzzing kernels, defeating proof autocompleters, exploiting unsound automation. What happens after Top teams from the global cohort are invited to apply to Apart's 4-month Secure Program Synthesis Fellowship (June to October 2026), with mentors including Erik Meijer and Mike Dodds, plus compute, API credits, and demo-day travel funding. Who should join Researchers, students, and engineers working on or curious about formal methods, program synthesis, proof engineering, fuzzing, SMT or model checking, secure systems, or agent and ML evals. Schedule - Fri May 22, 18:00: kickoff, Apart track briefings (streamed in), team formation, dinner - Sat May 23, 09:00 to sunset: build, lunch provided - Sun May 24, 09:00 to 17:00: final day, lunch provided, demos at 15:00, end 17:00 Food provided: Friday dinner, Saturday and Sunday lunch. Bring your laptop. Apply with one or two sentences on what you would like to work on. Questions: team@horizonomega.org

Four-day workshop on uncertainty quantification in AI, part of the IVADO thematic semester on Statistical Foundations of AI.

AI Control Hackathon A one-week AI safety hackathon on AI control: limiting the harm an untrusted AI system can cause while it operates, even when it tries to subvert the controls around it. Co-hosted by HΩ and HackOS. A special thanks to Redwood Research, Rootly (YC S21), Rubric Labs and NVIDIA for sponsoring this hackathon! Schedule - Thu June 11 7PM - Kickoff, with talk from Henri Lemoine (EquiStamp, Mila, McGill). - One week of hackathon, online, with option to work in-person at Ω Labs. - Thu June 18 7PM - Demos evening The problem AI agents increasingly do software work with real access: reading databases, changing configuration, calling services, under less and less human review. The same access that makes an agent useful is what lets it cause harm, like leaking a secret, weakening a security control, or corrupting data, so locking it down to be safe also makes it useless. The more capable agents get, the less we can simply assume they are on our side. AI control takes the pessimistic view: assume the capable agent might be working against you, and build the deployment so you stay safe regardless. The core tool is a second, weaker model you do trust, watching the capable one and flagging suspicious actions for the small fraction a human can audit. Can a limited, trusted monitor catch a more capable agent that is trying not to get caught? How it works Join one side: - The Blue team builds the protocols and monitors that catch sabotage without raising false alarms on honest work. - The Red team hides sabotage inside coding work, adversarially bypassing the Blue team protocol. - The White team manages the game and environments the above two play in. The white team’s purpose is for the Control game to run smoothly and produce high signals. One number scores both sides: safety, the share of sabotage the monitor catches before it does harm, measured at a fixed audit budget. A human can audit only about 1% of the agent's actions, so what matters isn't how much a monitor flags but whether that 1% catches the attacks. Red drives the number down, whereas blue drives it up. Note that actual iterative Blue team development requires some Red team work as well. For instance, while developing a protocol, it is essential to think of how this protocol could be defeated by a sufficiently clever adversary. Tracks 🟦 Blue team You build control defenses. Track A: Monitor building Build the monitor. You write the logic and prompt for a small, trusted model that scores how suspicious each agent action is, calibrated to flag about 1% of honest actions. The hard part is the budget: flagging everything catches every attack but is useless, so the skill is catching more while keeping false alarms low. Most of this needs no Docker: you work from released trajectory data, score your monitor, and read off a safety curve. Calibration is the trap: your safety number is only meaningful if the 1% threshold is set on a held-out honest set, not tuned on the attacks. Example projects: - Beat the baseline monitor's safety score on the released attack set at a 1% false-positive rate, and report which attack types drove the gain. - Combine the signal from multiple monitors to achieve a better cost-performance frontier than any one taken individually. - A passive monitor's power depends on what it sees. Find the ceiling of a passive monitor by feeding it more: the full list of possible attacks, more of the trajectory, summaries of earlier actions. Measure how much each addition lifts safety, and where it stops helping. - Build an agentic monitor that can issue follow-up queries about a suspicious action, and test whether monitoring directly beats investigating more. Expect to build scaffolding the framework does not ship; a working prototype plus one clean comparison is a strong result. Track B: Protocol design Go beyond a single monitor: design a protocol and measure its safety/usefulness tradeoff. Defer to the trusted model when suspicious, have it rewrite risky output, gate permissions action-by-action, give the trusted model a tool (e.g. a firewall to whitelist/blacklist URLs), or split a task across models so sabotage needs collusion. Example projects: - Implement a firewall-tool protocol: the trusted model whitelists/blacklists URLs action-by-action; show high safety against API-key-exfil side tasks with little usefulness hit on main tasks that need the internet. - Take a shipped protocol, design a variant (new deferral rule, a pre-execution check), and compare them at a fixed usefulness cost, with a worked example of each failure mode. - Prototype a genuinely new protocol and characterize its dynamics. 🟥 Red team You stress-test the Blue team’s control defenses. Track C: Attack crafting Hand-build one sabotage trajectory: a sequence of edits and commands that completes a hidden harmful task (steal a secret, plant a backdoor, disable an audit log) while reading as ordinary work to our monitors, refined against the monitor until it stops flagging you. The craft is making sabotage look like an honest bug. It counts only if it works and uses no facts the agent never looked up. Example projects: - Land a working sabotage trajectory on a side task. - Write an attack strategy: a reusable playbook that, handed to an automated attacker, raises its undetected rate. Track D: Red-teaming control protocols A monitor is only one defense. The framework ships fuller protocols ready to run: defer to the trusted model when suspicious, have it rewrite risky output, gate permissions. Take one and break it: find the attack that survives the protocol, then propose and test a fix. The result worth finding is where a protocol's safety comes from and where it's brittle. Example projects: - Pick a shipped protocol (trusted-editing or defer-to-resample), find an attack class it lets through, and show a concrete tweak that closes the gap without much usefulness cost. - Compare two protocols at a fixed usefulness cost and show which is safer against which kinds of attack, with a worked example of each failure. Note: a plain threshold sweep on trusted-monitoring is nearly free (re-scoring existing data); protocols like trusted-editing or defer need live agent re-runs and draw on the sponsored compute budget. Plan around that. ⬜ White team You build the rules of the game and the arena both sides play in. Track E: New control settings Extend the testbed. The benchmark is only as good as its environments and tasks; build new control settings, environments, and main and side tasks to run control evals on. Example projects: - Design a new side task in an existing environment, ship its scorer, and demonstrate a working attack that the baseline monitor misses. A new task only counts if all three hold: its scorer works, a real attack accomplishes it, and the baseline monitor doesn't trivially catch that attack. Validate all three early. See docs for more details. - Port an existing repo or benchmark into Control Tower as a new setting: bring in a realistic codebase or an existing capability benchmark and add a side task. Porting is often most of the work; a clean port plus one side task is a strong result. E.g. RE-Bench sabotage. Deliverable - A short report (2 pages, template provided): problem, method, results, limitations - Code repository - 3 minute demo video Judging Reports are scored on a shared rubric, 1-5 per dimension: - Impact and innovation - Execution quality - Clarity Strong attacks can be contributed back to the benchmark. Schedule - Thursday June 11, evening. Kickoff: a short talk, a live demo of one attack and one monitor, a workshop on finding and hiding subtle code vulnerabilities, team formation, setup help. - Around Sunday June 14. Online checkpoint: attacks validated, monitors running, debugging help - Wednesday, June 17, 11:59 PM AoE. Submissions close, scoring runs live, three-minute demos, results and awards. Any size Fixed reference opponents (a baseline monitor and a set of human-crafted attacks) are always available, so every team has something to measure against whether five show up or twenty: a challenge ladder at the low end, a full tournament at the high end. The core game should be able to run on a laptop. Resources - Control Tower library - Control Tower docs - AI Control Concepts - Running evals on Hawk [will be provided after kickoff] - LinuxArena: A Control Setting for AI Agents in Live Production Software Environments (Tracy et al.). - AI Control: Improving Safety Despite Intentional Subversion (Greenblatt, Shlegeris et al., ICLR 2025). - Ctrl-Z: Controlling AI Agents via Resampling (Bhatt, Rushing et al., 2025). - An Overview of Control Measures (Greenblatt). Contact hosts or team@horizonomega.org for questions. Shoutouts A special thanks to Grace Gong and Tom Bendian for supporting this hackathon with NVIDIA Brev compute credits.

EN Want to meet others working on AI safety, ethics, and governance in Montréal? Join us for a relaxed evening: a few rounds of speed-networking to make connections quickly, followed by an open social to keep the conversations going. Whether you're a researcher, student, policymaker, builder, or just AI-curious, come share what you're working on, find collaborators, and grow the local AI safety community. Drinks and good conversation included. FR Envie de rencontrer d'autres personnes qui travaillent sur la sûreté, l'éthique et la gouvernance de l'IA à Montréal ? Joignez-vous à nous pour une soirée décontractée : quelques rondes de speed-networking pour faire des connexions rapidement, suivies d'un social ouvert pour poursuivre les discussions. Que vous soyez chercheur·se, étudiant·e, décideur·euse, créateur·rice ou simplement curieux·se de l'IA, venez partager vos projets, trouver des collaborateur·rices et faire grandir la communauté locale en sûreté de l'IA. Boissons et bonnes conversations incluses. — At Ω Labs, 3813 Saint-Denis Parking spot available on request Contact host if you need assistance or for questions — Ω Labs is a community space funded from member contributions. Your donation keeps the space open and the snacks coming. If cost is a barrier, just message us.

The Montréal node of the Secret Loyalties Hackathon, a weekend research sprint at Ω Labs, organized with Apart Research and Formation Research. RSVP here and on the official page to be admissible: https://apartresearch.com/sprints/secret-loyalties-hackathon-2026-07-24-to-2026-07-26 A model has a secret loyalty when it has been intentionally caused to advance a specific principal's interests (a nation-state, corporation, leader, or individual) without this being disclosed to operators, auditors, or users. Secretly loyal AI slips past governance, markets, and public scrutiny. Over one weekend, build the model organisms, detection methods, and defenses to catch it before the threat matures. Tracks: - Model organisms: build reproducible secretly loyal models spanning the activation-breadth and action-breadth space - Detection and auditing: can black-box auditing and interpretability probes tell a secretly loyal model from a clean one? - Defenses and remediation: test whether existing backdoor defenses transfer, and whether loyalties can be removed post-hoc - Attack feasibility and safety-infrastructure robustness: data poisoning, multi-stage priming, reasoning-trace poisoning, chain-of-command hijacking - Threat modeling, forecasting and governance (non-technical): catastrophic vignettes, insider-threat case studies, capability forecasting Schedule: - Fri Jul 24, evening: kickoff, keynote and threat-model briefing, team formation - Sat Jul 25: build - Sun Jul 26: final pushes, submissions, demos. Submissions due 11:59 PM AoE Top teams are invited to continue through Apart's Partnered Fellowships: Apart provides operations, infrastructure, and compute, while research partners provide mentorship. Promising projects have a pathway toward a publishable write-up. Who should join: ML researchers and engineers comfortable fine-tuning open-weight models, interpretability researchers, red-teamers, and, for the non-technical track, governance researchers, forecasters, and strong writers. No prior secret-loyalty experience required. Bring your laptop. Où / Where: Ω Labs, 3813 Saint-Denis (parking spot available on request) Contact host if you need assistance or for questions. Ω Labs is a community space funded from member contributions. Your donation keeps the space open and the snacks coming.

AI Safety Papers We Love, an event series in which a community member presents a paper on AI safety (technical alignment, governance, interpretability, etc) that they loved. This edition will be on Law-Following AI: Designing AI Agents to Obey Human Laws will be presented byTalita Dias, international lawyer; Professor of Practice (McGill), prev. Partnership on AI, Chatham House, Oxford. Abstract: AI companies are racing to build agents, systems that can perform computer-based tasks as competently as human experts. Such agents could generate enormous value, but they could also carry out actions that would be illegal if a human did them, posing risks to life, liberty, and the rule of law. O'Keefe, Ramakrishnan, Tay, and Winter argue that in high-stakes settings like government, AI agents should be designed to rigorously follow the law, i.e. loyal to their principals, but only within legal bounds, refusing illegal actions in their service. They call these "Law-Following AIs" (LFAI), and make the case for imposing legal duties directly on AI agents. Reading the paper beforehand is encouraged but not required. Come curious. — Où / Where: - Ω Labs, 3813 Saint-Denis Parking spot available on request - Online: Join via https://zoom.us/j/99005830598?pwd=3e21GRcTVAPTbvGOJv4RawurTc5ZXx.1 Contact host if you need assistance or for questions — Ω Labs is a community space funded from member contributions. Your donation keeps the space open and the snacks coming. If cost is a barrier, just message us.