I highly recommend this great panel on Artificial Intelligence and International Development. Full of insights about how AI may change both more and less than you think. Starts at 5h36m45s of the video. #ABCDE2025

My experience with ChatGPT Agent so far: I've failed to find any use cases that _cannot_ be handled by Deep Research and yet can be successfully completed by Agent without running into any stumbling blocks like janky web forms or access restrictions. I'm sure I'll find some uses, but it will end up being a small fraction of tasks that come up in my workflows. If this is the case, it won't make sense to try to do new tasks using Agent unless it's a task that I would otherwise spend hours on (or would need to repeat on a daily basis). If my expectation is that Agent will succeed with a 5% probability, and it takes 10-20 minutes of trying painfully hard before giving up, it's not worth my time to even find out if Agent can do it. I would only use it if I somehow already knew that it's a task that Agent can handle. Given all this, I continue to think that task-specific agents will be more successful for the foreseeable future.

Quick thread on the recent IMO results and the relationship between symbol manipulation, reasoning, and intelligence in machines and humans:

🧠New on the CITP Blog from PhD student Boyi Wei (@wei_boyi) of the POLARIS Lab: "The 'Bubble' of Risk: Improving Assessments for Offensive Cybersecurity Agents" Read about how adversaries can adapt and modify open-source models to bypass safeguards. 👇

Back in grad school, when I realized how the “marketplace of ideas” actually works, it felt like I’d found the cheat codes to a research career. Today, this is the most important stuff I teach students, more than anything related to the substance of our research. A quick preface: when I talk about research success I don’t mean publishing lots of papers. Most published papers gather dust because there is too much research in any field for people to pay attention to. And especially given the ease of putting out pre-prints, research doesn’t need to be officially published in order to be successful. So while publications may be a prerequisite for career advancement, they shouldn’t be the goal. To me, research success is authorship of ideas that influence your peers and make the world a better place. So the basic insight is that there are too many ideas entering the marketplace of ideas, and we need to understand which ones end up being influential. The good news is that quality matters — other things being equal, better research will be more successful. The bad news is that quality is only weakly correlated with success, and there are many other factors that matter. First, give yourself multiple shots on goal. The role of luck is a regular theme of my career advice. It’s true that luck matters a lot in determining which papers are successful, but that doesn’t mean resigning yourself to it. You can increase your “luck surface area”. For example, if you always put out preprints, you get multiple chances for your work to be noticed: once with the preprint and once with the publication (plus if you’re in a field with big publication lags, you can make sure the research isn’t scooped or irrelevant by the time it comes out). More generally, treat research projects like startups — accept that there is a very high variance in outcomes, with some projects being 10x or 100x more successful than others. This means trying lots of different things, taking big swings, being willing to pursue what your peers consider to be bad ideas, but with some idea of why you might potentially succeed where others before you failed. Do you know something that others don’t, or do they know something that you don’t? And if you find out it’s the latter, you need to be willing to quit the project quickly, without falling prey to the sunk cost fallacy. To be clear, success is not all down to luck — quality and depth matter a lot. And it takes a few years of research to go deep into a topic. But spending a few years researching a topic before you publish anything is extremely risky, especially early in your career. The solution is simple: pursue projects, not problems. Projects are long-term research agendas that last 3-5 years or more. A productive project could easily produce a dozen or more papers (depending on the field). Why pick projects instead of problems? If your method is to jump from problem to problem, the resulting papers are likely to be somewhat superficial and may not have much impact. And secondly, if you’re already known for papers on a particular topic, people are more likely to pay attention to your future papers on that topic. (Yes, author reputation matters a lot. Any egalitarian notion of how people pick what to read is a myth.) To recap, I usually work on 2-3 long-term projects at a time, and within each project there are many problems being investigated and many papers being produced at various stages of the pipeline. The hardest part is knowing when to end a project. At the moment you’re considering a new project, you’re comparing something that will take a few years to really come to fruition with a topic where you’re already highly productive. But you have to end something to make room for something new. Quitting at the right time always feels like quitting too early. If you go with your gut, you will stay in the same research area for far too long. Finally, build your own distribution. In the past, the official publication of a paper served two purposes: to give it the credibility that comes from peer review, and to distribute the paper to your peers. Now those two functions have gotten completely severed. Publication still brings credibility, but distribution is almost entirely up to you! This is why social media matters so much. Unfortunately social media introduces unhealthy incentives to exaggerate your findings, so I find blogs/newsletters and long-form videos to be much better channels. We are in a second golden age of blogging and there is an extreme dearth of people who can explain cutting-edge research from their disciplines in an accessible way but without dumbing it down like in press releases or news articles. It’s never too early — I started a blog during my PhD and it played a big role in spreading my doctoral research, both within my research community and outside it. Summary * Research success doesn’t just mean publication * The marketplace of ideas is saturated * Give yourself multiple shots on goal * Pick projects, not problems * Treat projects like startups * Build your own distribution

Experts on body dysmorphic disorder have warned that people struggling with it have become increasingly dependent on AI chatbots to evaluate their self-perceived flaws and recommend cosmetic surgeries. "It’s almost coming up in every single session,” one therapist tells me.

If we compared AI capabilities against humans with no access to tools, such as the internet, we would probably find that AI already outperformed humans at many or most cognitive tasks we perform at work. But of course this is not a helpful comparison and doesn’t tell us much about AI’s economic impacts. We are nothing without our tools. And yet, many predictions about the impact of “AGI” are based on hypothetical human-AI comparisons in which the humans have internet access but no AI access. This kind of comparison is equally irrelevant. The real question is humans + AI vs AI alone. In such a comparison, AI isn’t going to outperform human-AI pairs, except in narrow, computationally heavy domains like games where speed is paramount and having a human in the picture only slows things down.* So whether or not AI will replace humans comes down to factors beyond accuracy — things like accountability, the ability to handle unknown unknowns, and potential preferences of customers and other workers to interact with a human, all weighed against the cost of employing a human. This is not to say that AI won’t displace jobs. But looking at capability benchmarks and going straight to claims about job loss is completely naive. * There are many studies where workers incorrectly override AI too often, but that’s because they received no training in when to override and when not to, which is an essential skill in AI-enabled workflows.

SB1047 was a bad idea. But Sen. Wiener's latest SB53 is on the right track, and it's important to call out the progress. Here's my reasoning. My approach to regulating novel technology like models is: we don't know how to define "good" mitigation and assurance, but we'll know it when—and if—we see it. There are two implications. #1. We shouldn't prescribe risk thresholds or standards of care for model development. We can't agree on the risks that matter, how to measure them, or how much is too much. The only guidance for developers, regulators, and courts is a set of nascent practices determined primarily by closed-source firms relying on paywalls to do the heavy lifting. Doing so could chill open innovation by exposing developers to vague or heightened liability for widespread release. That was SB1047 in a nutshell, along with ~5 equivalents it inspired across the US this session, such as the RAISE Act in NY. We should avoid that approach. These proposals are—in narrow but crucial respects—too far over their skis. And yet: #2. We need to shine a light on industry practices to better understand the diligence, or lack thereof, applied by different firms. If developers have to commit to a safety and security policy, show their working, and leave a paper trail, we can better assess the strength of their claims, monitor for emerging risk, and decide on future intervention. That is the EU's AI Act and final Code of Practice in a nutshell, which both OpenAI and Mistral have endorsed, and it's @Scott_Wiener's latest version of SB53 too. If we're going to regulate model development, that is fundamentally the better approach: regulating transparency—not capabilities, mitigations, or acceptable risk. It would give at least one US jurisdiction the oversight authority of Brussels, and it would avoid unintended effects on open development. To be clear, there are still icebergs ahead: > Complexity. Big Tech or not, these are onerous documentation and reporting obligations. Tactically speaking, the more complex, the more vulnerable this bill will become. > Incentives. Mandatory public reporting of voluntary risk assessments creates a perverse incentive for developers to under-test their models, and turn a blind eye to difficult risks. Permitting developers to disclose their results to auditors or agencies rather than publicly may help to promote greater candor in their internal assessments. > Trojan horse. California's hyperactive gut-and-amend culture can make it difficult to vet these bills. If SB53 morphs into a standard of care bill like SB1047 or RAISE, it should be knocked back for the same reasons as before. The more baubles are added to this Christmas tree, the more contentious the bill. > Breadth. The bill casts a wide net with expansive definitions of catastrophic risk and dangerous capability. For a "mandatory reporting / voluntary practices" bill, they work. If this bill was a standard of care bill, they would be infeasible. In sum: hats off to Sen. Wiener for thoughtfully engaging and responding to feedback over the past year. It's refreshing to see a bill that actually builds on prior criticism. There are still many paths this bill could take—and it has evolved well beyond the original whistleblowing proposal—but the trajectory is promising.

For folks wondering what's happening here technically, an explainer: When there's lots of training data with a particular style, using a similar style in your prompt will trigger the LLM to respond in that style. In this case, there's LOADS of fanfic: 🧵

In the 18th century, there was a real chance of death at any point in life, and there wasn't a big peak in old age. It wasn't just higher infant mortality - the whole distribution was completely different. Great chart by @Scientific_Bird.