#189 - Chat.com, FrontierMath, Relaxed Transformers, Trump & AI

Thank you.

Hello, and welcome to the last week in our podcast. We can hear a chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's most interesting AI news. And as always, you can also go to lastweekin.ai, our text newsletter for even more AI news we won't be covering here. I'm one of your host, Andrei Kerenkov. My background is that I finished a PhD focusing on AI at Stanford and I now work at a generative AI startup.

And I'm your other host, Jeremy Harris, co-founder and CEO of Cloudstone AI, the ICP National Security Company. I guess that's, I mean, we've said that many times, but now, now you really, really know. Yeah, how many episodes have you done now? It must be approaching 100. It's started. It's almost two years now. Yeah.

Right. You're right. We've missed a couple, but I mean, it's got to be knocking on the door of 100. I remember when we started, it was like in the wake of chat GPT, or that's when I came on, we'd each been doing separate podcasts in the meantime, but yeah, just like all of a sudden, everything went crazy.

This week won't be too crazy. I'll do a quick preview of what we'll be covering. And no huge stories this week. We got some nice new features being introduced by OpenAI either Tropic and the business front. We got some stories of fun things opening eyes up to a few fun open source projects and models of this week. So I think that'll be interesting.

some research on interpretability and efficiency for small models. And then policy and safety will be maybe the most meaty section we'll be covering. Let's just say we will be covering the implications of Donald Trump's victory for AI. Yeah. And as always, talking a little bit about what's going on with China and hardware and US restrictions.

Before we get into the news, as always, do want to acknowledge some listener comments. We had a few on YouTube. I always like seeing those. One person did say they like the idea of a community or discord. So that's interesting.

I'm not gonna make recall yet, but if we hear a few more, you know, maybe we'll make it and we can chat about AI news on there. And Jeremy, we did have a comment saying that a person loved your take on meta and when he singer Waits regards it to national security, which I think, I mean, it was

Yeah, it was mildly spicy. By the way, I want to add just a little, a little modifier to that. So the context was like, you know, some Chinese companies were shown to be, or sorry, China, China, China was being shown to use and rely on meta's open source models as a kind of

floor to their capabilities. Very important. We've known about this for a long time, obviously. When I say we, I mean the world. And so I basically said, we're getting to the point where it's indefensible. I think one dimension, somebody just discussed this on Twitter with me. It was a really good tweet. And I think something we've talked about earlier on the podcast. But I wanted to resurface here. They said, you know,

The advantage of open source, obviously, is you could put back doors in these models and thereby use them as a national security asset, have China use Western open source models that have back doors in them that we can then undermine. I think there are a variety of reasons why I don't think that's what's actually going on here. I don't think Meta is actually doing this strategy for several reasons that we could discuss.

I think it would be interesting. I think back doors are going to be really hard to try now because unlearning is notoriously fickle and superficial. So I just wanted to call that out or I think an important kind of additional level of detail to flesh that out with. So there you go. You can append this to my rent in the last episode if you want.

There you go, a little more nuance fair, which is always good. And also shout out to a couple more reviews. One of them did say keep up alignment comments and even said that we are hitting the goldilog zone onto the existential risk talk, which I feel pretty proud of. I think that's really intense. A lot of work went into that. Yeah.

And we did have a critical review, which I appreciate calling out the intro AI music. It seems that not everyone is a fan, terrible, truly terrible AI generated songs from the intro, which I don't know, I like them, but maybe I'll keep them to like 15 seconds instead of 30 seconds. And as always, I'll put them at the end for people who do enjoy them.

And one thing before the news once again we do have some sponsors to give a shout out as with the last couple weeks the first one is the generator which is Bob some colleges interdisciplinary AI lab focused on entrepreneurial AI. Bob some college is a number one school for entrepreneurship in the U.S. and that has been the case for 30 years and just last fall professors from all across Bob some that

partnered with students to launch this generator, which is a lab that is organized into a groups such as AI, entrepreneurship and business generation, AI, ethics and society, and things like that. And it has now led peer training of

faculty all across Bobson. Their intent is just to accelerate entrepreneurship, innovation and creativity with AI. So yeah, it's a very cool initiative. We will have a link for you to check it out if you're curious.

And one new one actually, we do have a second sponsor and it is Darren McKee promoting his engaging AI safety book Uncontrollable. The full title of it is Uncontrollable with threat of artificial super intelligence and the race to save the world. So if you do like the AI risk talk, I think you might be

Interested in this book, Mark's tagmark, who you would know if you care about AI safety, said that uncontrollable is a captivating balance and remarkably up-to-date book on the most important issue of our time. It explores topics like uncertainty, control and risk.

And yeah, it makes the case that we should be concerned about advanced AI, but it's not a doomer book. It lays out a reasonable case for AI safety and what we can do about it. We'll have a link to it on Amazon in the show notes. And it's also on Audible. You can just search for it. The title is uncontrollable.

Yeah, I actually have had quite a few conversations with Darren on this topic too. So he thinks a lot about it. He's talked to a lot of people as part of his research for this book. So certainly if you're interested in that space, I definitely wanted to pick up and read, again, Max Tegmark. One out of one Max Tegmark's agree that this book is a book and a great book and maybe the best book probably. That's a little preview, maybe a risk on it.

All righty, now on to the news. We are starting as always with tools and apps. And the first story is about OpenAI introducing a predicted outputs feature. This feature can speed up GPU 4.0 by up to four times for tasks like editing documents or refactoring code.

So the gist is, many times when you're using LLM, you may only want to tweak your input. So you may give it some text or some code and say, you know, correct any grammar mistakes in this document, for instance.

And so that means that you're mostly going to be spitting out what you take in with just a few tweaks. And that is the gist of what this is. If you use this, then you can have much faster outputs. For me, it's actually a little surprising. It's taking this long for this feature to come out. It, I think, is pretty well established, something you can do. But nice to see both on Tropic and OpenAI introducing more and more of these

really develop a friendly, you could say, features. Yeah, this is definitely part of that productization push right towards more and more kind of application specific tooling that OpenAI is focusing on. You know, one of the things that is making this possible is speculative decoding. This is the technique that it's been around for a little bit now, but now we're seeing it productized. The basic idea behind it is you get two different models. You have a draft model, basically, it's like a very small cheat model. And at any given time,

You can get that draft model to propose like, what are the next five tokens or something like that, right? So get it to cheaply produce predictions for those tokens. And then what you can do is feed all five of those tokens in parallel to a larger model that would have more expensive computation, but it can handle them in parallel all in one forward pass.

spending the same amount of computers at what if it was just like one input that it was trying to process. And then you essentially get out predictions for how accurate the draft models' token proposals were. And so this allows you to essentially amortize the cost of that more expensive model over a large number of tokens, get it to do sort of editing and clean up, so to speak, a lot faster and a lot cheaper. So this is a practical implementation of speculative decoding.

It's one of those things where, you know, it's funny, you read the paper and then a couple of months later, it's like, boom, you know, people are putting in a production actually saving a lot of money. So this is the whole idea. Another advantage of course is you don't have the problem that the model might, you know, hallucinate the stuff that's solid. Like if you have, you know, some small part of like a JSON file or something that you want to tweak and you want the rest of the file to be anchored to be exactly the same, then this allows you to do that, right? It allows you to fix.

What they're doing during speculative decoding is they're actually here fixing the part of the output that should be fixed and only having the large and expensive model make those predictions presumably on the variable parts of that output. This is a bit of a janky reimagining of what speculative decoding looks like with this added constraint that the stuff before and after this window that you're actually going to try to sample in is concrete is locked in.

I think kind of cool. I'm curious about the economics. What they are doing, by the way, is they're only charging you for the tokens that are actually getting kind of modded in the middle, let's say, wherever you want the modifications to occur. So that seems fair, right? You're giving a strong prior on keep the beginning and the end, say, the same. So don't charge me for generating those tokens. Only charge me for generating the ones that I care about, which, again, makes a lot of economics sense.

That's right. And there was also, I guess, a partnership of factory AI with OpenAI to test the same feature in their API. And they have a few metrics. It's not like, you know, there's no benchmark that they report here, but they do have some numbers of they did find in practice two to four times faster response times.

while maintaining accuracy and they have examples of large files that would take 70 seconds throughout this taking 20 seconds roughly. So yeah, very easy to see how this is useful in practice for various applications.

Next up, we are moving to anthropic and a price increase for haiku 3.5. It costs four times more than the predecessor. The claim, I think, is that the price hike is at least partially because haiku 3.5 is superior to the previous version.

But rather surprising. So the new pricing is $1 per million input tokens and $5 per million output tokens. And that's, again, four times more than the previous one.

It's also almost 10 times more expensive than GPT-40 Mini. When we look at that, that's pretty remarkable. In fact, it's only two and a half times cheaper than the full GPT-40. GPT-40 Mini was supposed to be the haiku, say, of the opening eye series, of the GPT-40 series.

Here we have essentially a model haiku that's coming out and saying, hey, I'm still the small one, except I'm now going to cost you something closer to the full model size. That's a really interesting play, and I think this speaks to something very interesting that's happening with the economics of these models. One of the big questions has been, we've talked about it a lot here, but

To what extent do LOMs just get commoditized right to the point where the margins go to zero? Your model is basically the same as their model is basically the same as your other competitors model and so everybody has to just basically price based on the raw cost pretty much of producing the model and serving it.

And at that point, your profits go to zero, or this is kind of what happens economically. And one of the challenges is you can't do that and build enough bank to spend for the next generation of massive multi-billion dollar data centers if you're just living a hand to mouth existence like this, so a bit of a structural problem.

This is the first time we've seen that trend bucked, where we've seen a model come out and say, hey, you know what? On the basis of my higher quality, I'm going to up the cost associated with using this model. A lot of developers, some might say fairly understandably, are coming back and saying, hey, this is an unwelcome development, not necessarily because of the price increase per se, but because the framing

is that, hey, this is better quality, so therefore we're charging you more. This is really interesting, right? There's this classic thing when you do startups, when you do, I get me, it's more broad than that, it's economics, really. When you're trying to sell something and make a profit,

value-based pricing is the thing you go with, right? You advertise how much value you can bring to the customer, how good your product is, rather than talking about the cost. When you talk about how much it costs you to make a thing, that's a hint that your whole industry has been commoditized, right? So when you go out to McDonald's and you say like, hey, well, can you give me the same burger just to buck cheaper? They'll tell you like, no, the patty costs this much, the bun costs this much, the cashier's time costs this much. So therefore I have to sell you this, they probably won't do that, they'll probably tell you leave, but whatever.

They'll literally tell you, sir, this is a Wendy's. Anyway, but you kind of get it, right? When you're dealing with a commoditized industry where everybody can basically offer you the same product, your margins go to zero, you argue based on cost. This is different. Claude and anthropic is coming out and saying 3.5 haiku is higher quality, therefore will charge you more.

People pushing back on that is an indication that, well, actually, this space is pretty commoditized. Anyway, I think this is a really interesting tell. One of the big consequences, by the way, of all this stuff, as you see prices going up and down and side to side and you've got new products coming online,

It really makes a lot of sense if you're a company working in this space to have the scaffold you need to very quickly assess through automated evaluations whether the task you care about is being performed well by a given LLM.

So a new LLM comes online with a new price point. You should be able to very quickly and efficiently assess, does this LLM at this price point, at this quality, make sense for my use case, right? If you can't do that, then you can't ride efficiently this wave of lower and lower LLM prices. You're not going to benefit from that in your product. So just a kind of, I guess, side thought there, you know, really important for companies to get into the habit of checking these latest models, because there are companies for whom haiku 3.5 is going to be way

way better than the other options. But the question is, what are you competing against? Are you competing against GPD 4.0 or are you competing against GPD 4.0 mini? And right now we're somewhere in between. This is, yeah, I think to me a little surprising. The announcement of 3.5 haiku was at the same time as 3.5 sonnet, which recovered, I think, about two weeks ago now. And it was just this past week that they announced surprise change. And that is what led to people responding. And four times,

Raise a price is pretty dramatic. So it must be a mix of it was underpriced to begin with, perhaps significantly underpriced. And I guess there's also perhaps a factor of them just emphasizing 3.5 sonnet as the main one they want to compete with going forward. I don't know. Yeah, it's certainly an interesting move from a competitive perspective.

On the lighting round, we are starting with Flux 1.1 Pro Ultra and RAW. So Flux 1.1 Pro from Black Forest Labs, one of the leading AI image generator providers, has now been upgraded to support 4x higher image resolution up to 4x

million pixel, I don't know what MP stands for, but really high resolution and it still has faster generation times of 10 seconds per sample. And this is priced at just six cents per image. And they do have this raw mode as well, which just leads to kind of realistic looking images more akin to photography.

So, I guess not too surprising, we keep getting better and better models, more and more realistic, but I think we're keeping up with Black Forest Labs and they're moving pretty rapidly in the space. Yeah, and they're the ones who have memory servers partnered up with X.

You know formerly known as Twitter to support the grok app in the image generation functionality that they're developing so You know this is this is them continuing to pump out their own independent product line which I don't know maybe integrated as well with with grok at some point Yeah, looking at the images again, I mean I find myself continually saying this I'm not an image guy so I like I don't know the you know the the kind of

The aspect of image generation, let's say, of greatest interest to people who really dig into the space. But the images look like they're really high quality. The raw mode especially does look really gritty and real. Because I'm a bit of a buffoon in the space, I kind of look at these and go, cool, I feel like I've seen a lot of other models that have the same quality. So I'm kind of not sure where the

where the mode is in this space, but still does look impressive and Flux has kind of come out of nowhere, too, with these new models. And speaking of X and Grock, we have a bit of a story on that. X is testing a free version of a Grock chatbot in some regions. So this was previously exclusive to Premium and Premium Plus users of X.

And now there is a free tier where you can do 10 questions in two hours for the group, two models, and 24-week rock to mini models, plus a few image analysis questions per day.

So you do have to sign up to X, of course, and you do need to have a linked phone number. But certainly, you know, this is something that you have in chat, GPT, I think also in anthropic, the ability to use the chatbots for free. So

This is just being tested in New Zealand now, but it would be interesting to see if a continue of expansion to more users. Obviously, a big goal anytime you launch something for free like this is to collect user data, uploads and downvotes for, say, RLHF or something else. Also, just to own more Mindshare, I think one of the things that OpenAI continues to enjoy a massive lead on is the fact that chat GPT is a household name.

Whereas Claude is not. And Grock increasingly is becoming one, but that's only thanks to the distribution they get through x. And so I think at this point, you combine the x distribution factor with the x factor, if you will.

with the fact that this is free. That could be really interesting, but the code is interesting too, right? Like a query quote of 10 questions within two hours. I don't know about you, but when I'm sitting down with like with Claude, which is for some of the work that I do, I tend to spend quite a bit of time with Claude, actually.

They're long sessions, and there's a lot of back and forth, and there's a lot of like going back and editing questions and, you know, tweaking prompts. So that quota might be challenging for some of the heavier use cases, which makes sense. Yeah, this feels like, you know, you want to give people a taste. Yeah, exactly.

People might consider subscribing to X, which hard to say. I'm not sure if Grock will convince people who aren't subscribers to do so, but maybe. No, you're right. I mean, there's value in bundling it on with X, right? Like I was going to say, there are other free chat platforms that don't give you a limit, but the fact of the X integration, that distribution is so, so key, and I think it's still probably being underrated, so we'll see.

We have two applications and business speaking of chatbots. We have a kind of fun story, not a very consequential story, but one that is neat. OpenAI has acquired the domain chat.com. We don't know the exact details of how much will cost, but it appears to have cost a lot in the maybe

10 million-ish region. We know that it was previously acquired by Hopspot co-founder Dharmesh Shah for 15.5 million, I think just roughly two years ago or so. And it has now been revealed that he sold shoutout.com to OpenAI. And Sam Ottman

on X tweeted or posted just chat.com. That was the entire post, I guess, showing off. So it's not yet, I guess, been promoted heavily. There's no new brand. It's so-called chat GPT, but, you know, I mean, 10 million for UL. That's pretty significant.

Yeah, I mean, if it were 10 million, that would be a haircut on the initial acquisition cost of 15.5 million, which is pretty significant. But from the context, it seems like something more interesting maybe going on here. It seems apparently

Apparently, as if Dharmesh Shah, the guy who acquired it, may have been paid an open AI share. So if that's the case, that would be kind of interesting too. He had this somewhat cryptic post on X. All of this is very cryptic. It's the most cryptic launch of a new domain I've ever seen. But if you do go to chat.com, you will see, of course, the, right now, the chat GPT-40 interface. So there you go.

Right. Yeah. To emphasize 10 million, we don't know if that even is of a ballpark that's just based on what was previously paid. You would expect it to be, you know, around that maybe. Next up, a more serious story. And it is that Saudis are planning 100 billion AI powerhouse to rival the UAE tech hub. So the Saudi Arabia, of course, and it's planning this artificial intelligence project to

Yeah, pretty much develop a technological hub to rival that of the United Arab Emirates. This will be used to invest in data centers, startups, and other infrastructure. It's titled the initiative project. It's called Project Transcendence, which is pretty fun. Not too stupid at all. Yeah.

Well, you know, pretty ambitious, you could say. And of course, this will also be used to recruit talent to the region, which I'm guessing is perhaps not quite as prevalent there as in the US or elsewhere.

So, yeah, we've covered in the past how the UAE has invested significantly. There have been developments from the region like with Falcon models that were pretty notable at the time. Don't know that we've had too much to cover in recent times from the UAE, but certainly it's true that these countries are trying to invest and be a player in the space.

Yeah, I mean, I think the biggest kind of recent stuff with the UAE has been infrastructure, kind of structural stuff with G42 and the questions around, you know, can they decouple from Huawei technology in Chinese tech and, you know, the Department of Commerce getting involved there. So really the question about where is the future of, say, AGI training run scale data centers? Where is that going to be? And this idea that the UAE has this massive energy advantage, which is a big part of the reason and capital, which is a big part of the reason why so many people

are interested in it as a hot bed as a place to build out the infrastructure. This is Saudi Arabia basically saying, hey, wait a minute, we're a giant oil rich nation with deep concerns over how much longer that oil is going to hold up and be viable.

And so they're looking for ways to diversify out of that industry and well guess what oil comes with the other awful lot of energy and that's great so it gives them a lot of the ingredients they need again the money and the energy to potentially see something like this they already have.

Similar structures, let's say, to project transcendence. There's a company, a sort of state-backed entity, called Owlet. It's a fund that does sustainable manufacturing. It's got $100 billion in backing. That's about the order of what's speculated, could be associated, could be associated with project transcendence. We don't know yet how much actually will be forked over, but there are discussions with potential partners, which include, I think I saw Mark, or sorry, Andreessen Horowitz. Yeah, that's right.

So apparently, A16Z is talking with the public investment fund, which is sort of the state kind of entity that would be overseeing all this. So that's kind of interesting. I mean, a Western private actor looking at that, apparently, the fund itself is maybe growing to as large as $40 billion in commitments, again, aiming for that 50 to 100

billion in total, which would be pretty impressive. But keep in mind, that is about a year of Microsoft infrastructure spend. And the challenge here is that the build-out for this is slated for 2030. There are a whole bunch of problems right now plaguing Saudi Arabia on this front as well. They've seen an overheating economy that's now causing the clawback, some of their previous commitments, to do similar build-outs in other tech sectors too, including semiconductors and smart

you know, smart everything basically. So, you know, now there's a little bit of uncertainty about the future of some of those projects. This one certainly has a lot of buzz around it. So, you know, see where that ends up going. And by the way, because it did a little digging,

What kind of history does Saudi Arabia have in the LLM space? I was not tracking this, but there was a 7 billion parameter model. It's the only one I've been able to find so far, but for, you know, take it for what it's worth. There's a tech company called Watad that apparently built this model called the Muhlhem, and it was a Saudi Arabian domain specific LLM that was trained exclusively on Saudi data sets. So a bit of a scaling issue there in terms of getting beyond that, but

There you go. So they have a small footprint in the space, obviously hoping to attract talent, which is going to be a really, really important resource. And I think that that's going to be a challenge for both Saudi and, frankly, the UAE as well, at least on the model development side, the infrastructure side, I think, might be a bit of an easier play.

Yeah, so good call out there. This is saying a backing of as much as a hundred billion and this is a people familiar with a matter kind of article. So yeah, not the many concrete details there.

After the lightning round, the first story is again on OpenAI, but this time it's about hardware. It's that Metas former hardware lead for Project Orion is joining OpenAI. So this is Caitlin Kalanowski, who was the former head of Metas AI glasses team and has also worked on VR projects.

and also worked on MacBook hardware at Apple is now joining OpenAI, seemingly to focus on robotics and partnerships to integrate AI into physical products. We covered pretty recently how OpenAI did start recruiting for robotics positions with the descriptions of a job having to do with integrating charge PPT into robots.

We did see a figure, the developer of a humanoid robot showcase their robot working with shadubt, having conversations and being told to do stuff. So perhaps this is this recruitment points to opening. I want you to do more of that.

Yeah, there's a lot of reading tea leaves, especially this week with OpenAI and its hires. Apparently, part of the speculation in this article is that Kalanowski is there to work with love from this. Her old boss, Johnny Ive, we've talked about Johnny Ive partnering. He was the designer, of course, of the iPhone.

Now he's been brought on board to open AI to launch as he put it a product that uses AI to create a computing experience that is less socially disruptive than the iPhone. So I couldn't quite interpret what he was saying there is was he saying it's going to be less less horrible socially than the iPhone was or it's going to be less of a.

Game changer in the iPhone was probably he met the former. I'm not sure. But anyway, so apparently she'll be, you know, back working with him. So that's sort of a natural, a natural partnership there. She has a lot of experience doing design and Apple as well. Really, really unhelpful, I will say, of open AI to have two separate media threads that involve the word Orion because there's this model we will talk about, right? The speculative model we talked about this for.

of the model Orion, and now you have the former Orion lead from Meta, different things coming to OpenAI. I really wish that they would keep their headlines a little bit straighter. Yeah, why Orion is a, you know, be a little more original, okay, in your project names. Also worth mentioning, OpenAI did acquire company building webcams earlier this year, I believe. So, could we play into that? We don't know. It is just, we don't know what they're doing here.

It's also interesting about face, right? Because they disbanded their entire robotics team. This is like four years ago, and now they're really rebuilding it. But it does seem that the new robotics team is a lot more market focused, like product focused. And so that in itself is sort of interesting. There are pros and cons there. They'll get a lot more real world feedback by having their systems out there and more interesting data. But yeah, anyway, so the structure of OpenAI continues to tilt towards more and more of a product oriented work.

And just one last story on OpenAI. This one is, I guess, a fun one as well. And it is that OpenAI accidentally leaked access to the upcoming 01 model to anyone by going to a certain web address. So this was accidentally leaked in the sense that users could access it by altering a URL.

For a brief period of time, it was shut down after two hours, I think maybe when people wear it or something. So we have the preview model of 01 that you can use, but still we don't have access to the full 01 version. Now, yeah, people were able to play around with it.

opening actually confirmed that this was the case and said that there was not too much access since this was resolved. So people play around with it and as you might expect did say that it was pretty impressive.

Yeah, the opening, I at least said that they were preparing with a limited external access to the opening and model and ran into an issue. So I guess in the process of trying to give people, you know, maybe special links to access it. It leaked in that way.

I still think, so by the way, some of the demos are kind of interesting. There's a classic one where you have this like image that is an image of a triangle and it's subdivided with a whole bunch of lines. And then those lines form sub triangles with the image. And then you ask how many triangles are there in the image? Standard multimodal LOMs really struggle with this.

In fact, the preview version of 01 struggled with this and got the answer wrong. The new version did not. So one of these little things where maybe a bellwether eval or something like that, who knows. But I think one of the most interesting aspects of this, apart from the fact that it teaches us quite a bit about opening eyes continued struggles with security, it must be said.

you know this is this is an organization that uh... explicitly has said that they are trying to prevent people from seeing the full reasoning traces of all one because that is critical intellectual property for them um... well guess what this uh... this all one version the full one version which was leaked to begin with

also leaked out a full chain of thought when it was asked to analyze, in one case, a picture of a recent SpaceX launch and other things in other cases. For this sort of critical competitive secret, really, and that's what it is, the reason opening, I didn't want to release those chains of thought initially, was precisely because they were concerned that those chains of thought

would be really valuable training data for people to replicate what is so precious about this model series. And so, you know, here they are kind of leaking it out themselves with this, you know, haphazard launch. So it doesn't really inspire a lot of confidence in opening eyes security approach their philosophy, really, frankly, the level of effort that they're putting into this. I know it sounds like a small thing, but when you're dealing with, you know, this is stakes as they may potentially present themselves in the future, national security otherwise, like this is not a small screw up.

It could have been mine. If you imagine it's not an individual who's accessing this, it's an AI agent or something, and it's collecting, using the opportunity to collect a bunch of training data, not saying you could do a ton of it in that time, but this is an important vulnerability. A little disappointing, especially given that OpenAI has made such a big public show of trying to get into the security game more.

And just one little caveat with regards to the full chain of thoughts. We don't know for sure if that's the case. One Twitter user reported seeing it, but that may or may not have been the full full. It was just a detailed response that did include some of the reasoning steps.

Yeah, I know that's fair enough. It did look different enough. Yeah, you're right. It did look materially different enough from the standard reasoning trace that's put out. And similar enough to the one that the reasoning traces that were shared that opening, I did share right when they launched that it's like very suspiciously like it seems like at least it's similar to what it's doing internally. Yeah. Yeah.

And one last story and video is once again even more valuable than before. This time it is the largest company in the world. It has surpassed Apple on Tuesday. I don't know what happened on Tuesday. I guess we'll find out. So the shares rose at 2.9% leading to a market capitalization of 3.4%.

3 trillion ahead of Apple at 3.38 and for Defence Microsoft is at 3.06. For reference, NVIDIA has gone up by more than 850% since the end of 2020.

So yeah, still an insane story of Nvidia's rise. It's sort of funny because it's like all my friends at the labs, like not to make it a whole stock story, but a very, very big wave of people who went in hard on Nvidia from the frontier labs, like in the sort of like 2021-2022 era.

You think about the revenues they're making, plowing it into Nvidia, and now that's kind of 10xed in value. Anyway, there's a conviction about where this all might be going. We're not giving stock advice on the show. Don't invest based on our stock advice. But certainly, eye scaling has been good to Nvidia.

Yeah, I will say I remember when I was in grad school, like in 2017-2018, and I was like, oh, wow, NVIDIA is really doing good because of all this deep learning stuff. And their GPU is being the backbone of deep learning, which is the big thing in AI. And even at the time, I was like, I wish I had money to invest, and it was not a poor grad student. And Jensen saw that in 2014-2013, right? He has been positioning NVIDIA and the whole CUDA ecosystem for this for a long time.

And yeah, it's pretty wild. Moving out to projects and open source, the first story is about new research, which we've covered a couple of times, and VEM launching at user-facing chatbot. So this group has previously released Hermes, specifically Hermes-370B in this case.

It's a variant of META's LAMA 3.1, and noose research. One of our big trademarks is these unrestricted models, so having free access for all, doing completely unrestricted ability to track, so less safety guardrails.

This one, the article writer at least, did find that it did refuse to do certain things, like go into how to make drugs, although according to a new service is not from them. So they didn't add any guardrails to this user-facing chatbot. Some of it was already baked into a model previously.

Yeah, I do find this interesting. There's a certain eagerness to do fully, fully no guardrails. I don't think even XAI doesn't, or sorry, even the platform X through Grock and XAI, therefore, they don't pretend to be trying to do a fully no holds bar thing, right? They're like, we will adhere to the law.

and not produce things like child pornography or whatever else. So same things happening here and noose is interesting because they are especially into this thesis. What I interpreted earlier is like a more extreme way, but here they're basically saying like, oh no, of course we have safeguards on the actual model. Of course, we try to prevent it from doing really, really bad things like helping you make illegal narcotics like meth naturally.

So anyway, the model, as you'd expect, has been jailbroken. Pliny the prompter. Very quick on the case, as usual, finding a really powerful exploit that basically gets them through everything. It's so interesting. I mean, I'd love to do a deep dive on Pliny the prompter's methodology and approach, because there's some fascinating stuff there.

New really interesting to note that they're even launching this right this is not a new model it is just a chat interface so they are trying to play in that space as well. Yeah so we'll see where it goes i mean i i don't know if they're going to be charging for the stuff at some point or how that'll play out but they are really into the make it available for everybody up to including training methodology right we covered their distro.

Optimizer a couple episodes ago that anyways meant to make it possible for like people to pull off massive training runs distributed across basically the whole world between GPUs that type thing. So anyway, there it is. That's right. And this is I suppose part of our platform news chat. So that's very much like charge you be TV interface. You log in, you have a text prompt window. It has a fun kind of

visual style to it a little more like, I don't know, old windows or a terminal. It looks a little, I don't know, nerdy, I will say. And one fun thing about it that is kind of interesting is you do have access to a system prompt and you can modify directly, which is not the case.

with Chad GPT. So just to read a bit, the system prompt that is here by default is your Hermes and AI to help humans build, create, flourish, and grow. Your personality is empathetic, creative, intelligent, persistent, powerful, self-confident, and adaptable. You communicate it formally, and it's a sick responses that feel just like an other human, etc, etc.

I don't know, neat that they do provide access to that and you can configure it. Next up, we got Frontier Math, a new benchmark. So this one is crafted by over 60 expert mathematicians from top institutions and has original unpublished problems across various branches of modern mathematics, meaning that you shouldn't be able to find it on the web and learn out that

So compared to existing benchmarks like GSM 8K and MAF, which have simpler problems and do have the benchmarks out there with a test set, here you have problems that require deep theoretical understanding and creativity. And as a result, things like GBP4, Gemini 1.5 Pro struggle and solve less than 2% of the problem. I believe that was

quote from Terence Tao, one of the people involved that they should be challenging for models who are at least a year or at least a couple years. Yeah, and they've got an interesting framework that they, so it's not just the benchmark, right? They're coming out with a whole evaluation framework.

that's all about automated verification of answers. Part of that is to prevent guessing. They want to prevent LLMs from being able to succeed just by throwing out a guess and doing well. They set up these questions so that the responses are deliberately complex and non-obvious, like the correct answers to reduce the chances of guessing, getting you to where you want to go.

They're also designed to be the kinds of problems that it wouldn't take like, it's not just a question of, you know, I want it to take me a really long time to find the answer to this question, but I can do it through relatively straightforward reasoning, right? So it's not like an undergraduate physics question, for example.

It's also not like a, some of the, you know, GPQA questions, like the graduate question answering questions, which sometimes you can answer in one shot. Like without thinking, you need to have the expertise, but if you have it, in some cases in that data set, you can just go ahead and respond without thinking too, too much. They're trying to combine those two things together. They want it to be like really, really hard and also require

hours, if not days, as they put it, of human thought time to solve for. So you can really see. Everybody keeps saying this with new benchmarks. If a model can solve this, then it's going to be AGI. The only AGI will be able to solve this. The problem is every time we keep seeing these new benchmarks come out, there keeps being a trick. Some way to make models that do really, really well at it. Occasionally, those tricks actually have broader implications for AGI, for the spillover in general knowledge.

But that can happen quite often, but they certainly don't require the full kind of AGI that some people think they might. So this one, yes, we're at 2% right now, success rates for cutting-edge language models like Cloud 3.5 Sonnet, Gemini 1.5 Pro, all that stuff.

Yeah, unclear what's actually going to get them there. Is it a better agentic scaffold? Is it a better trained foundation model? What is it? It's going to be interesting to see what actually ends up cracking this metric. Pretty impressive to see, or at least a sign of the times, you could say, that

now people are developing these absurdly difficult things that most humans could even try. They have some sample problems. This one that is in paper, they say, is high to medium difficulty. Just to read the problem, it's construct a degree 19 polynomial p of x in c of x such that x

has at least three but not all linear irreducible components over x choose p of x to be odd monic have real coefficients and linear coefficient negative 19 and calculate p of 19. So I don't know what that means and the solution they provide in paper is like an entire page.

Yeah, a bunch of references to various theorems and so on. So this is like hardcore math over here. And I suppose it's not surprising that current albums can't beat it just yet. Yeah, and you can really see in that problem phrasing there, the layering on of sequential requirements that makes it harder to guess, right? You can't just like one shot that even with a guess, like you'd have to guess multiple things, right? Which reduces the chances that you get a

and anomalous results. So it's all meant to make it automatically evaluateable. Geez, having a hard time saying the words. And last up, we do have a new open source model. This is Hunyun Large, an open source mixture of experts model with 52 billion activated parameters by Tencent. So this has 389 billion total parameters.

And it's pretty beefy and impressive, so it can process up to 256 tokens, 256,000 tokens. And does it beat llama 3.1 70b on various tasks, like logical reasoning, language understanding and coding seems to be

somewhat comparable to llama 3.1, 450, 405b. So certainly seems like Tencent is trying to flex a muscle and showcase stability to build this scale of model.

So one of the interesting things about this paper is so they present a whole bunch of scaling laws. And they share their thoughts about how many tokens of text and of text data and how many parameters and all that. So when you do the math, at least by my math, which Claude is very helpfully helping me with, we get to compute budget about 10 to the 21 flops. And compute budget is always something that

It's good to be interested in when you see a Chinese model because one of the things that they're really constrained by is US export controls on hardware. And they find it really hard to get their hands on enough hardware to train these models. So here we have 10 of the 21 flops. So for reference, when you think about a GPT four class model, alumna, alumna three 400 B class model, you're looking at training budgets there of about 10 to the 25 flops.

So we're talking 10,000 times bigger than this model in terms of compute budget. So I find this really weird. They claim that this model is on par with llama3400b. I may be missing something in my calculations. If somebody, if you can spot this, please do. This seems to me to be very much stretching. This seems very frankly implausible. I must be missing something or the paper must be missing something.

But if that is the compute budget, then they're doing something really janky, really weird. And that would be the headline, like if the actual compute budget was that. But again, yeah, llama 10,000 times greater training budget. And here they're saying that it performs on par with llama 3.1405B. So that doesn't make any sense to me. Would love to. Yeah.

It seems that maybe there's a typo, maybe we didn't quite run the equation right. They do say they trained for 7 trillion tokens and there are 59 billion activated parameters that would mean that it shouldn't be that different on that order of magnitude.

Lots of details in the paper. They do talk about the architecture, the number of layers, the attention heads, the type of the tension used, which is also a case of llama. So these kinds of details on the nitty-gritty of how this is implemented always, I think, is useful for pretty much everyone working on LLMs.

And now to research and advancements, we begin with some work from Google and some affiliates called relaxed recursive transformers, effective parameter sharing with layerwise LoRa. So this is a pretty novel, pretty interesting technique for getting more out of tiny models. As we've seen, we've made

More and more gains in the space of one and two billion parameter models and this one introduces the notion of recursive models. What I mean is they train like a vanilla transformer has n layers right in each layer is distinct.

What we do in this paper is say that you can take a set of layers and then basically stack them again and again. So you have repeat layers a few times in a row. And just by doing that, you're able to still go to small size, but retain the performance of a larger model. And that's per the title of the paper. The relaxed part there is that

Well, they do repeat the layers a few times. They still apply Laura to differentiate them slightly across layers. So that, I think, is a neat little technique showcasing continued progress in the space of being able to really squeeze out all the performance out of less and less parameters.

Yeah, this is a really interesting paper for a lot of reasons, including the hardware interaction here. But for sort of intuition building, I found this really weird way when I read it, to be honest, I was like, how I wasn't familiar with the literature around, because there is some around, I guess, what they're calling recursive transformers, people have done some little experiments, right?

And then actually just to call this out, it might be confusing. So recursive is going back a little bit. It has been research on recursive, different from recurrent. So recursive is different because you're not kind of updating a hidden state. There's no like time sequence element here. It's really just you have one input and you pass it through the same neural network several times.

to get through a better ad. So you take an input, you pass it through weights, you get an output, you put that output back through the same set of weights. And that's what it means to be recursive. And yeah, it's been out for a little while that it actually is possible to train neural nets to be better after several recursive passes, several passes through itself. And yeah, I'll let you Jeremy take over.

Yeah, no, but that fact itself, right? That's something that I was not aware of going in myself, and it struck me as quite counterintuitive, right? You feed the same data. You put data into a model, a layer one, and then you make it go through layer one.

And then instead of going to layer two, you may go back through layer one and over and over and over again, and you get a better result out. And I was trying to build some intuition around this. Best I could tell is like, so reading a book twice, right? You're kind of doing the same thing. Even though you're using the same algorithm or the same layers and all that, you're able to extract more and more information with each pass. And so this is essentially the same principle, basically. You're chewing on the data more,

You can think of it as a way of just expending more compute to in the process of chewing on that data if you want to compare it to just like feeding it through just the layer one time, now feed it through multiple times. You get a better result. So one of the challenges is, sorry, let's talk about the advantages. First, the advantages, you are copy pasting the

same layer over and over, which means you don't need to load a, I don't know, an 8 billion parameter model. Maybe you get to load a 4 billion parameter model if you reuse every other layer, right? Or anyway, you could keep playing games like that. Where do you have layers stack like three times in a row, the same layer, and then a different, you know, the next layer and copy that one three times in a row. Or it could be all the same layer. There are all those configurations that are possible. And so

One of the advantages here is it cuts down on the amount of memory that you need to use on your chip, right? This is really good for memory usage. Still need to run the same number of computations, though, even though your layers are identical, your weights and parameters are identical, your data, the embeddings of that data are changing, so you still have to run those calculations. So the logic costs, the number of flops, the flop capacity of your hardware, still needs to be utilized intensely.

There is a way that you can even get an advantage on that level, though, because so much of your computation looks the same, it makes it easier to paralyze it. So they have a section of the paper on continuous depth-wide batching where they're talking about, okay, how can we leverage the fact that the layers are identical to make the actual logic?

Less demanding on the chip which is which is really cool But the really big boon here is for memory usage because you're literally you're functionally cutting down on the size of your model right pretty dramatically So that's that's really cool. It's it's such a dead simple method There is this technique that they're using that seems to work best in terms of deciding which layers to copy paste that they call their stepwise method This was the the one that worked best. So basically they would take a

If you have, I don't know, like a 20 layer transformer, they would take every other layer and copy it once. So you take layer one, repeat layer one, one time, right? Then take layer three, which would be the next one, because you layer one, layer one, then layer three, layer three, then layer five, layer five, layer seven, layer seven, all the way up to 20. And that's kind of the thing that they found worked best. The intuition behind that, just being that, hey, there were like,

There was prior work that showed that this worked. So a lot of this is just sort of like janky engineering, but still a really interesting way to kind of play with, again, play with hardware, see what can we do with chips that have crappy memory, but maybe good logic. You know, unclear like which chips would necessarily fit into the category once they use this continuous step-wide batching strategy, but really interesting and a great way to get more out of your AI hardware.

This paper has quite a bit to it, a lot of details that are interesting, so they do use the step-wide strategy initially, but when they add this other trick of Laura for these layers to be able to adopt them slightly for different layers,

they do a site modification to a subclass method where they average two layers so like layer one is an average of layer one and four and then the other one is an average of 2.5 just empirically they found this work better and you do need to they say up train so you need to train an initialized model for a little while to get it to work well

But we do say that you don't need to train it very much, with just like 15 billion tokens of up-training. A recursive Gemma1 model outperforms even full-size pre-trained other models like Pfea and TinyLama. So yeah, it's quite interesting and we'll be seeing a guess if this gets adopted in practice.

And I don't know if we talked about the Lora adapter role in this kind of conceptual structure, but maybe just worth emphasizing, when you lock in these parameters and you're just repeating the same layer over and over, you might want to give your model a little bit more degrees of freedom

to kind of adapt to the new problem domain that you're going to be training it on. And that's really where the lower adapters come in. It gives the model a little bit more room to stretch itself, right? Hence the relaxed qualifier here and relaxed recursive transformers. You're giving it a few more degrees of freedom to kind of modify itself without that constraint of all these layers have to be the exact same. So that's kind of the intuition.

Yeah, right, so Laura also, for some references, a way to efficiently change a bunch of weights by tweaking a smaller set of parameters. You could basically reduce it too, so that's the idea here is you're not updating, you're still sharing most of the weights, but you update a few parameters that make them a little more distinct.

And on to the next research, we got applying a golden gate cloud mechanistic interpretability technique to protein language models. And this is not a paper, actually, this is more of an open source project that looked into the idea of applying the same technique that we've covered, I believe, like now a few months ago.

where we had sparse auto encoders, where that can be applied to LLMs to get interpretable features. So you can say the famous example, I guess, is the Golden Gate Bridge.

feature in Claud, you can see that there is this kind of notion concept within Claud that gets activated for certain inputs. And that is done via the sparse autoencoder technique that compresses the outputs of certain layers in LLM and then finds regularities at a high level. So this work was applying that same technique

to a model specialized in protein prediction, I guess, protein language models. And they found some interesting features in this context. And I think, Jeremy, you read more into this. I'll let you go ahead and take over.

I really like this paper. And for context too, the SAE, the sparse auto encoder is a bit of a darling of the AI interpretability world, especially among folks who care about loss of control scenarios and is my AI trying to plot against me trying to scheme as the, believe it or not, the technical term is. So the idea here is, yeah, you have somewhere, so let's pick a middle layer of our transformer.

And we'll pick specifically the residual stream. So residual stream is basically the part, the circuit in the machine say circuit. The part of the architecture that takes whatever the weights were from the previous layer, just copy paste them into the next one. It's a way of preventing the information from degrading as it gets propagated through the model. But anyway, essentially pick a slice of your transformer and you feed the model some kind of input.

and you're going to get activations at that layer. Now, pick those activations and use them as the input. You're going to feed them to a model called their sparse auto encoder. The sparse auto encoder is going to take those activations and it's going to have to represent them using a small set of numbers, like a compressed representation.

So, you know, maybe you have, well, as a cartoonish version of the say you have 10,000 activations, then you want to compress them down to like a hundred dimensional vector, right? So that's what the sparse auto encoder is doing. It compresses them. And then from that compressed representation, it's then going to

decompress them and try to reconstruct the original activations. And the loss function it uses is usually something like the difference between the old and the true and the reconstructed activations. So basically it just gets really good at compressing these activations down to a smaller representation. It turns out, and anthropic found this, that when you do that, the individual entries in that smaller compressed representation end up correlating to human

interpretable features. For example, the idea of deception might be captured by one or a small number of those numbers. The idea of a molecule might be captured in the same way. This is basically just meant to be a way of taking this very complicated thing, all the activations in this residual stream, and compressing them down to a manageable number

Of of of numbers that we can actually get our arms around and start to interrogate and understand and interpret right so that's kind of part of that hope of the alignment. Game plan is like we'll be able to use this to understand the thinking real time of a eyes that are very potentially dangerously advanced that's the theory.

A lot of interesting success has been found there, including on steering the model's behavior. If we do something called clamping, we pick one of those numbers in that compressed representation, and let's say it's the number that represents banana or encodes the idea of banana, we crank up its value artificially, and then we reconstruct the activations. We can then get the model based on those activations to generate outputs that are tilted towards banana, whatever that means. Maybe it talks a lot about bananas or something like that.

That was the Golden Gate Claude experiment, right? So they found the entry that corresponded to the Golden Gate Bridge, they clamped it to give it a really high value, and then that caused the model to yap on about the Golden Gate Bridge. And so here are the questions going to be. Will we find the same thing?

if we work on transformers that are trained on bio sequence data and they pick a model that was developed by this company ESM, sorry, this company EvoScale that's made the ESM series of models. So we covered ESM three many months back, fascinating model. It was the first ever bio sequence model, by the way, to meet the threshold of reporting requirements under Biden's executive order back then. So it was a really, really big model. What they did was they took a smaller model, ESM two that that company had built.

And they play the same game. Can we pick a middle layer of that transformer, build a sparse auto encoder, and can we recover human interpretable features? Can we find features that correlate with, in this case, a common structural components or facets of biomolecules? A common example here would be the alpha helix.

So if you put proteins together, certain kinds of amino acids together, certain kinds of amino acids, when you string together to form a protein, they'll tend to form a helical structure called an alpha helix. The other secondary structure that they sometimes form is called a beta sheet or beta pleated sheet or whatever. They're all these different

structures that these things will form depending on the kinds of Lego blocks, the kinds of amino acids that you string together. They all have slightly different charges, so they attract and repel in these nuanced ways. It's notoriously hard to predict what the actual structure is going to be. Well, here, using this technique, they're able to find, okay, we actually have in our SAE in that reduced representation, we have some numbers that correlate with, oh, there's going to be an alpha helix here, a lot of alpha helices.

or beta hairpins or whatever else. That's interesting from an interpretability standpoint. We can understand a little bit more what goes into making these proteins take the shapes they do, but then they also found that by modifying the values in that compressed representation, by doing this clamping thing and artificially being able to

let's say we enlarge the value of the alpha helix number, you could actually prompt the model to output sequences that would have more alpha helices. This is interesting from a protein design standpoint. It's the first tantalizing hint, well, maybe not the first, but bucket it with alpha fold as a series of tools that could allow us to better understand

how proteins fold and actually come up with designer proteins with certain structural characteristics that otherwise would be really, really hard to design.

And on to the lighting round, we begin with a pretty fun blog post from nap time to big sleep using large language models to catch vulnerabilities in real world code. So this is by Google's Project Zero. And this is a team that's been around for a while since 2014, dedicated to finding so-called zero

vulnerabilities in code that aren't yet known or out in the wild. That hackers can then exploit without there being protections for it. They have previously had this project nap time.

valuing offensive security capabilities of large language models. They had this blog post several months ago, where they introduced a framework of large language model assisted vulnerability research and demonstrated the potential for improving state of our performance on the cyber, sec, eval to benchmarks from meta. That was a little while ago. And now nap time has evolved to big sleep, where Google Project Zero is collaborating with

Google DeepMind. And in this blog post, they announced a pretty exciting result from this big sleep agent with LLM that's optimized for helping with, I guess, vulnerability detection. They discovered vulnerability via this agent, an unknown real vulnerability in a major project, SQL Lite.

and reported it, and the developers fixed it. So to their knowledge, this is the first time NAI has been used to find a real world vulnerability like that. And this blog post goes into a whole lot of detail, into vulnerability, which seems to be a somewhat tricky case, not some sort of trivial discovery, so to speak. So very exciting for implications of being able to fight hackers with AI.

Yeah, and also a warning shot that, hey, these things can actually, AI can now discover real-world vulnerabilities. It's always a double-edged sword with these things, but yeah, and that's been a big question mark, right, in the debate over AI and what risks it might pose is

You know, I've had debates with people or, you know, they'll say, well, you know, we haven't seen an AI system actually successfully discover cyber vulnerabilities in real world systems. And so therefore, et cetera. Now that we have, I mean, I wonder, I wonder what the implications may be, but there have been

Pilot studies we've talked about a couple finding first it was one day vulnerabilities where the exploit has already been, been logged somewhere and now you're just getting in a agent exploit it and then zero days which is really figuring out without knowing, whether there is even a vulnerability finding one from scratch in kind of more toy settings this is the real world though this is finding one in a, i mean sequel light is a.

a very, very popular library. And this is an interesting bug, an interesting exploit. It's a null pointer dereference, which essentially is you have a pointer that points to memory addresses. And this vulnerability allows you to control what it points to. And so this allows essentially to have some control over what gets written or read to memory. And that could, could in principle, allow the attacker to pull off arbitrary code execution.

And essentially, if you just point the pointer to some specific buffer space or some adjacent memory, you may be able to actually draw that, pull that data in and use it for whatever purposes. So besides that, they're just making the application crash. You have to have a fucked up pointer or something, and it just won't work.

All that interesting, they go into how this thing works and it is quite an interesting improvement over current techniques like the best techniques we have right now, which include things like fuzzing where you basically just throw everything in the kitchen sink at your application, at your software, and just see if anything breaks. This is a much smarter approach, obviously, powered by a thinking AI system. Pretty cool.

This was, by the way, a bug that did remain undiscovered after 150 CPU hours of fuzzing. People had tried the standard techniques on this many times over. Makes sense, it is a popular library, but those techniques failed, whereas this AI-powered one succeeded.

And one more story intersection. This one not about progress, but rather lack of progress and some unknown research. So it's about OpenAI has been a report from the information about them reportedly working on new strategies to deal with an AI improvement slowdown.

So OpenAI has been working on something like a GPT-5. The upcoming model has been codenamed Orion, and there you go. That's the reference to Orion from before. And the report is saying that it seems to not be showing as significant an improvement over predecessors as in previous iterations.

in the leap from GPT-3 to GPT-4, that was a massive improvement. GPT-3 was pretty impressive. GPT-4 was much more impressive. And GPT-4 now was, oh, I don't know, like two years old.

No a year and a half old it's been a while since GP4 came out and we haven't had that kind of leap since Except maybe you could argue with 01 with the introduction of inference time compute we saw some pretty significant qualitative gains regardless

This report from the information is saying that the sort of commonly used the standard trick of more data, more compute, more scale may not be as effective as it was previously. So, of course, we are having a scarcity of new training data. That's one of the issues is most of the internet has already been sucked up.

And the surveyors is reportedly this new foundation steam within OpenAI looking at possible alternatives to just scaling, like doing more in the post-training phase, doing more with synthetic data from AI models, et cetera.

Now, OpenAI has not commented on this and has previously said they have no plans to release Orion or anything like GP5 this year. So, can take it with a grain of salt, but also maybe not super surprising.

Yeah, I think this is such an interesting part of the debate and the question over scaling, right? So there's a question as to whether... So when we look at scaling curves, what we're typically talking about is how well does roughly the model's next word prediction accuracy improve with more data, more compute, right, and model size?

What the challenge is, that doesn't tell you how that improvement in next word, sorry, next word prediction accuracy does not necessarily tell you how generally useful a model is, how good it is at reasoning, other things that we might actually care about. And so you've got this very robust scaling law that tells you that the model is getting better predicting next tokens.

but then uncertainty about the value that's being created in that process. That's one dimension of uncertainty, without knowing what's going on with Orion, what the training data looks like, what it's intended to do. Is this another reasoning system? It seems like it's not supposed to be, but there's a lot of fog war here.

without knowing that it's hard to know whether what I've just described here is an important part of the uncertainty or whether it's like a reasoning model and the inference stuff isn't working out. From what I've seen, it seems like it is more likely to be the former that this is really meant to be a beefy, pre-trained, you know, GPD-5 type model as opposed to 01, which was, you know, putting, I mean, I really don't want to say bells and whistles. It's way more than that, but it certainly is leaning more towards the inference time paradigm. And that was, that's the big leap.

there. We have separate inference time scaling laws now to write as well that complement the training time scaling laws. That may well be enough to do some really interesting things. There's a whole bunch of interesting gossip about OpenAI in here. Apparently, back when Orion had only completed 20% of its training run,

Sam was really excited about it and was talking internally about how this would be a big deal. That's where he hyped it up. It seems that that hype has failed to materialize. That's really what's kind of an issue here. There's also questions about what hardware this stuff is being trained on. What is this training run? I'm guessing it's the H100 fleet that opening eyes running right now to train this. What scale? What are they really pushing in terms of scale? Really hard to know.

And just more generally, because they are setting up this foundations team to explore deeper questions now, if the default path to scaling, the engineering path will call it, where you just build faster horses. If that doesn't work, what do we do instead? That's the big question. I think in this instance, OpenAI is really, and quite ironically, put itself in a difficult position over the last few years. They've bled off, I think it's fair to say,

all of their best or not all, much of their best algorithmic design talent, right? So, Iliya Setskiever has left. We've seen, you know, the safety team, you know, we've seen, anyway, like, basically, a huge, huge amount of talent, including product talent, we had Barrett Zawfully recently, too. There's, like, really, really good folks who are gone, in many cases, to anthropic.

And so if it is the case that we're moving from a domain where it's about exploiting a paradigm, in other words, doing really good engineering and getting scaling to work really well to a paradigm where we're looking for new ideas instead, where that's the main bottleneck, then you might anticipate talent being the main limiting factor in which case anthropic

starts to look really interesting right you've got a lot of companies now that could be that could be competing here meanwhile opening eyes hamstrung by a relationship with Microsoft that is currently tepid at best in the recent investor communications Microsoft did not refer to opening eye in the future tense at all.

That is a big, big change. As that starts to happen, is opening eyes forced to work with companies like Oracle to develop infrastructure because Microsoft apparently isn't meeting their needs. There's tension there too. This starts to become really interesting for Sam. He's got to find a way to square this circle. He's got to find a way to keep raising money. He's got to find a way to keep scaling for what that's worth. Then he's got to retain talent.

It would be interesting if this turned into a very significant structural challenge for OpenAI, if they've doubled down too much on scaling. But again, this is all speculative. We don't know until the models start dropping. And frankly, I think when the Blackwell series of GPUs come online, we get those big clusters running next year. I mean, look, everybody I know in the space really expects big performance improvements from the early tests they're doing.

I suspect we'll be looking back on scaling is like, yep, that was a real thing all along. But if not, the implications for opening AI, at least are interesting. That's right. And also, if not, English is not sort of unique to open AI, right? It's an open question in general. If it is even doable to scale, in part, because of training data running out, that was a speculation for a while. And just paint a high-level picture, right?

What scaling means is GV3 was around 180 billion parameters. GV4 we don't know, but the speculation of the rumors, where it was around closer to 2 trillion total parameters, but it was a mixture of extra models.

some smaller set of activated parameters. And so, GB5 or whatever the next model is, Orion, you could say maybe would have 10 trillion total parameters, or 20 trillion, you know, that kind of jump in size and a speculation is, well,

if you do that same kind of move from GB3 to GB4 to GB3 and just add more rates, add more scale, add more training, will you get that bigger jump? Right now it's unclear, right? And this report is basically claiming or seems to claim that maybe it's not quite as successful as it has been in the past, but it remains to be seen.

Yeah, I think worth noting, though, on the data scarcity side, there is eventually a data wall, presumably, unless synthetic data carries us over. The data wall, though, is expected to kick in about an order of magnitude of flops, like training compute, further than, for example, like power constraints. And right now, we're not even close to power constraints in our current runs.

10 to the 26 flop runs next year, probably shading into 10 to the 27. That's still like two orders of magnitude before you hit even the power constraints on the grid. So right now, I don't think that the data scarcity is actually the thing driving the limited capabilities here. I think there's something sort of

something else is going on here. And we'll presumably have to wait and see if that's part of the reason why I'm curious, what happens at that next beat when we get the Blackwell clusters online? When we start to see the 100,000 GB, 200 GPU clusters running, do you then see the transcendence to use the Saudi Arabian terminology for this? Do you start to see that kind of improvement? I don't know, but I think it's, yeah, there's a lot of

experimentation with many billions of dollars that will be run to find out.

Alrighty, moving out to policy and safety, and as promised, we are going to talk about Donald Trump's victory for the presidential election in the US, and in particular what it means for AI. No political commentary from us, even though as a citizen, I have some opinions. But regardless, Donald Trump is going to return to a White House.

And there's not a ton we know about specifics of what might happen, but we do have some pretty decent ideas as to at least some of what will happen. So for instance, we do know Trump's administration is presumably going to repeal President Biden's executive order on AI, which we've covered plenty on the podcast, which is a very big

Order, not a law, so the Trump administration, because this was just an executive order, the Trump administration could just cancel it, more or less. Now, there might be retention of some of the features of that. It might be revising it rather than fully canceling it, but it does seem likely, at least. We don't know for sure, but certainly there will be revisions to that.

And then, of course, we know that Trump loves to fight with China, and that's been an ongoing situation in the US for a while. So there will probably be more of that. But Jeremy, you're a policy guy, so I'll let you do more. So we're talking here. Yeah, I mean, I used to think of myself as a tech guy, I guess.

I guess half and half bad. Yeah, no, look, I think it's funny because in the, so the policy universe I live in is the national security policy universe to the extent that I live in the policy universe. And I think that there are a lot of people in the kind of

general AI safety world who are really concerned about a Trump administration. And I actually think that a lot of those concerns are quite misplaced. Like I think this is a misreading of what we need and where we are. So just for context, we've seen Trump on various podcasts. That's all we have to go on, by the way. And this article goes in depth into comments that Trump's made on

There's been no promises, no guarantees, so this is kind of reading to leaves and guessing based on various comments. Exactly, exactly. And so Trump has rightly described, in my opinion, AI as a superpower and called its capabilities alarming. He's also referred to China as the primary threat in the race to build advanced AI, which I think is also correct.

And then you've got this interesting question as to, you know, cabinet staffing like Elon is a massive influence in the cabinet and to the extent that that. Or I should say sorry on the transition team and brought down the team. I don't know that he'll be in the cabinet officially because he's kind of busy with a lot of company. But he, you know, obviously a massive influence, very concerned about.

Everything from weaponization to loss of control, a lot of good quotes from Dan Hendricks in this article who advises Elon quite a bit. And then the question is, that's Musk. That's Elon. You've got Vance on the other side, Trump's VP, obviously, who's expressed concerns in the past over a closed source AI entrenching the tech incumbents. Now, I mean, I think this is a, it's a very rational concern to have, right? Like you don't want closed source

pure plays and not allow people to open source stuff, I think that is going to start to change inevitably. As you start to see open source models actually getting weaponized, it's just going to become super obvious to all concerned. And at that point, the administration clearly is preserving their optionality to go in that direction at the time. Some big questions here remain around the AI Safety Institute, for example. That was sort of a spawn off of the executive order. A lot of the bones were laid there.

Interesting question as to whether that remains. It is the case that most Republicans do support the AZ. It's a part of the broader American strategy on AI, and it's certainly a home for expertise. The question as to whether Paul Cristiano continues to run it.

That's another degree of freedom they have. They keep the AZ, but swap out Paul Cristiano, who the former head of line that had opened AI, who invented reinforcement learning from human feedback, so that would be an interesting question. But then more broadly, the executive order, the famous Biden executive order, 110 pages, it was the longest EO in living memory,

I think there are a lot of components there that are likely to be preserved in a Trump administration. I think you'll see some stuff get scrapped. Look, that EO did tons of stuff. It talked about bias and civil rights and all kinds of stuff under the banner of AI. I think you could well see that get carved out, hollowed out.

Trump has said he's going to rip out the EO. That's not a question. That will probably happen. But what it gets replaced with is really an issue here. How much of the national security stuff gets preserved? I wouldn't be surprised if we end up seeing an awful lot of that stuff still in there. Anyway, there's all kinds of questions as well about what do we do on the energy infrastructure side.

We have a ton of work in the United States to do to get energy back on the table. We have forgotten how to build nuclear plants. We can't build them faster than 10 years. We need a way to keep up. We just talked about the power bottleneck and how that kicks in at about 10 to 29 flops. Well, that's coming. That's the training runs of two, three years from now. If it takes you 10 years to build a nuclear plant, then you've got to change something pretty fundamental.

We need to get natural gas online. We need to get geothermal potentially. And a lot of these things align with the kind of Trumpian school. So making sure AI gets built here. The questions are all going to be around, what about things like loss of control? What about things like weaponization and open source? Those are the big question marks. And right now, again, it's an administration that's positioned itself very openly, very flexibly.

You know, the China angle, I think, is a very bipartisan piece too, right? So I don't think we're going to see all the export controls that have put in get ripped out. I think those are actually going to be bipartisan and maintained where we might see a change would be the Trump administration may be focusing more on enforcement, right? We've covered a lot the leakiness of these export controls under the card administration would be great to see, you know, actual loophole closing as fast as loopholes are opening. And that's something you could see.

One last kind of meta note here, the uncertainty that we see around what might come out of a Trump administration here reflects uncertainty in the technology, but it also reflects the classic kind of Trumpian move of maintaining uncertainty for the purpose of negotiation leverage. You see this with tariffs and all the discussion around that. The threat has to be credible so that it actually leads to leverage internationally. It's something that we've seen

other administrations anyway struggle with is like, if you're speaking softly and you're not carrying a big stick, then people will not take you seriously. And to the extent that there's a lot of negotiation to do with China on this issue, you may actually want to negotiate from position of strength.

And for that, you need to have the energy leverage and other things. So I think big, big questions around the AZ, big, big questions around what the focus is on open source and on loss of control. But with Elon there, I think there's a lot of room for positive stuff to happen, potentially on the safety side.

So yeah, I think the story is, again, much more positive that a lot of the people who I know in the kind of AI safety world seem much more concerned about this. And I think part of that may just reflect a concern over the

frankly politics like some people are just they just don't want they don't want this administration and that's part of it but right now it's unclear and and you know just got to wait and see I think there's some really good policies that have been put forward generally on the energy side and elsewhere so wait and see is the best approach probably.

Right, yeah, that's generally my impression also this article goes into and basically does lay out the picture of that. It doesn't seem like there's any obvious big overturnings of what's been happening. There's going to be a lot of tweaks presumably similarly over chips act.

which was one of the major movements during the Biden administration. Trump has been somewhat critical of it, but it's unlikely that the Republican Congress and Trump will repeal that act. They might kind of arise it, but it does seem more likely that that will stay in place and continue being a factor.

Yeah, that's I guess this article is a summary in our best guess at the implications of a Trump presidency for AI. We will have to wait and see what happens in practice.

And speaking of evading AI sanctions from the US, the next article is FAB Wacamal, Chinese companies are evading US sanctions. And this is a bit of an overview, I suppose. So it's talking about the need for AI competitiveness, governing the sanctions.

And talking about how companies such as Huawei are exploiting various loopholes to acquire advanced semiconductor manufacturing equipment, which is then enabling them to build large AI clusters. So again, Jeremy, I'll let you take over on this one since this is your real house.

Oh, yeah. Well, I thought this. Okay. So, so I will always show a semi analysis. Any chance I get semi analysis is an amazing newsletter. If you're into AI hardware stuff or hardware stuff, I should say in general, go check them out. The blog posts are really technical. So unless you kind of know the

hardware space, tough to justify a subscription if you're not getting all the value out, but if you're in that space, I mean, you're probably already subscribed. These guys are amazing. So this is, yeah, a report on the really difficult enforcement challenges that are facing the Department of Commerce and BIS as they look to

enforce their export controls on AI chips. But I just want to give you an excerpt from this report. They're talking about SMIC, which is China's answer to TSMC, obviously. So they produce all of China's leading nodes on the hardware side. So they say sanctions violations are egregious. SMIC produces seven nanometer class chips, including the Kirin 9000 S mobile SOC system on a chip, and ascend 910B AI accelerator. Two of their fabs,

Okay, two of their fabs are connected via Wafer Bridge. Okay, so Wafer is the thing that you, it's like this big circular thing that is made of silicon and silicon. And that's what you etch your circuits in. And anyway, this is the starting point for your fab process. So two of their fabs are connected via Wafer Bridge such that an automated overhead track can move wafers between them. But for production purposes, this forms a continuous clean room and effectively one fab. But

For regulatory purposes, they're separate. One building is entity listed by the US. In other words, one building is owned by an entity that's on a blacklist. You're not allowed to sell advanced AI logic to them. And because of national security concerns, whereas the other one is free to import these like dual use tools and it claims to only run legacy processes.

And yet they're connected by this physical fucking bridge. This is how insane it is. You basically have one facility and we're just going to trust China and SMIC that they're not sending something like a wafer right when it should be going left type of thing. That's the level things are on. They go into detail on stuff that we've been tracking for a long time. So there is a fab network that is being run and orchestrated by Huawei where they spin up

new subsidiaries, basically as fast as they can to evade US export controls. Right now, US export controls work on a blacklist basis. So you basically say, OK, we're going to name new entities and organizations. You are not allowed to sell advanced semiconductor manufacturing equipment to. And we try to keep that list fresh. Well, Huawei is just going to kind of create new spawn new entities as fast as they need to.

And they have this vast network now that is basically moving Huawei into the center of what you might think of as China's maybe AI ambitions. Like if you start to think about what is the, not even the open AI of China, but what is the coordinating entity for a lot of China's big scale AI work, it is increasingly Huawei both on hardware and software. So there are all these pushes to get Huawei looked at in all this.

And what this report argues for, and I think is quite sensical, is you need to start to think about tightening in a broader way your export control requirement. So instead of just saying, oh, look, we've got a blacklist, and we're going to try to keep that blacklist fresh.

Instead, let's say a wider range of tools to require any material that is at all US fabricated in the whole supply chain, that that can't be shipped. So even if you're at ASML, you're building something that has any component of US technology in it. If you ship that to China, that's a no-no.

These broader tools are becoming necessary just because otherwise you're playing this whack-a-mole game that you're destined to lose. And at this point, the stakes are just way, way too high. So by the way, I say this, you know, semi-analysis is they are AI accelerationists in their bones, right? This is like they are not kind of AI safety pill. As far as I can tell, it's quite the opposite.

And here they are saying, no, no, we need to fucking ban the export of this hardware to China in a very robust and unprecedented way. I think this makes all the sense in the world. If you believe this is ultimately, you will use technology, then that's what you gotta do. We can't be updating Blacklist every 20 minutes.

And just a couple more stories. The next one is very much related to that previous one, actually an example of sanctioned violations. So the story is that the US has fined the company global foundries for shipping chips to a sanctioned Chinese firm. So this is

500,000 penalty on this New York-based company, Global Foundries. It's the world's third largest contract chip maker, and it has shipped chips without authorization to an affiliate of SMIC, the Chinese chip maker. And this was

74 shipments of 17.1 million worth of chips to this company SJ semiconductor, which is affiliated with SMIC. Interestingly, this also says that global foundries voluntarily disclosed this violation and cooperated with the Commerce Department.

And that was a statement from the Assistant Secretary for Export Enforcement, Matthew Axelrod says, we want U.S. companies to be hyper-vigilant when sending semiconductor materials to Chinese parties. And the global foundries came out and said they regret, quote, the inadvertent

action due to a data entry error made prior to the entity listing. So a data entry error blamed for this. Look, probably true. And this is the stuff is really difficult to enforce, especially when you have a very complex set of layered kind of requirements and all this stuff. Like, you know, the rules right now are not simple. And that that is a challenge for enforcement.

So maybe no surprise to see this is yet another kind of leaky situation. Obviously TSMC had similar issues recently. They accidentally sold some stuff to a Huawei affiliate. But this is just what happens. It's part of the reason why you just need stronger incentives.

friend of companies like Global Foundries are running processes that are subject to these kinds of errors, then that just implies, okay, they need to try harder. The incentives just need to be stronger. To kind of bump back to that semi-analysis report that we were talking about earlier, one of the callouts that they make is the industry started. This has been claiming that this would wreck

tighter export controls would wreck industry and blah, blah, blah. And they've actually been doing better, not worse, including decent sales to the Chinese market in the last few years. This has been an absolute boom time for them in spite of increasingly tight export control. So the economic argument may be faltering a little bit here, but yeah, we're seeing in real time these holes kind of appear. And yes,

get plugged, like this will get plugged, and then there are going to be new holes. It's this, yeah, never ending game of whack-a-mole again to plagiarize the semi-analysis post-title. And last up with Story is that on Tropic has teamed up with Palantir and AWS to sell its AI to defense customers. Quite related to the story last week, we had with Meta,

altering their license, their user agreement to let defense in the U.S. use it. Now, the discoloration would allow Claude, Vishal, but from an anthropic to be used within Palantir's defense accredited environment as Palantir Impact Level 6.

I don't know what this is, reserved for systems containing data, critical to national security. So Antropic previously has, you know, I guess, prevented use of Antropic or at least precluded it in their arguments for US defense customers. And Pervis article and Pervis what we discussed last week seems to be part of a general trend.

Yeah, anthropic I have heard has been really transparent internally with their own teams about this and the deliberative process behind it. I mean, I actually think this is, you want an AI-CT-focused org to be working with the US government to have them understand what's going on, including in defense context. And this is gonna be for, yeah, intelligence analysis, that sort of thing. So yeah, I mean, I actually think they're gonna face a lot of flack for this. I think this is a good move.

And the Palantir partnership is actually going to be really important for them too, because selling into DOD is hard. You want to work with someone who really understands that process. So, yeah, this is another big boon for anthropic potentially, because that market is also just, it's really big. And it's what anthropic needs to do to understand both their customer, their really big potential customer well, and also for their own mission, they need to

to be able to integrate tightly with the US government, with the national security parts of the US government, and all that stuff. So yeah, we'll see where this goes. And if we end up seeing more reporting about this deal.

Yeah, and speaking of the government, this news also covers that cloud has come to AWS GovCloud, which is a service designed for US government cloud workloads. Wasn't aware there was a GovCloud, but that's neat. So, seemingly, it's not just for military, it's also just in general for use within the US government.

And that will be it for this episode of Last Week in AI. Once again, you can go to the episode description for links to all the stories. Also go to lastweekin.ai for those links and for the text newsletter.

We always appreciate your comments, your views, your tweets, your comments, all those things, a button of anything. We do appreciate you listening, so please keep tuning in and hopefully enjoy this AI song. That is not terrible.