← Back to 2018 talks
[APPLAUSE]
Hi. I'm Tom 7. Someone told me it was possible to rise out of the stage, since we're in an actual opera house. So at least I got to kind of mimic that. But maybe next year.
So here's my talk. It is a weird talk, so I'm trying to kind of go for, when we vote at the end on weirdest talk, and most confusing talk, vote Tom 7.
And along those lines, I have to do this. Not really a joke. There's some flashing in this talk. It's sort of part of the nature of the work.
And so if you're sensitive to that, this talk may not be for you, or you may need to close your eyes or something. It gets particularly bad in the coda. So I'll try to warn you when that's happening. This slide is gratuitous. But some of the other stuff that's going on there is just part of what's going on.
But speaking of what's going on, who is this man in front of you? Who is Tom 7, if anything? I have done some things you may have seen. And I'm just going to show you these, A, because maybe it makes you think, oh, that guy; I'll listen to this talk. But B, because I think it's characteristic of the kind of work that I do, and my talk is a meta-meta joke about a project or something. So that's part of why it's so confusing.
So this helps you understand a little bit about where I'm coming from. By the way, I do have a normal day job where I contribute to society. So these are my hobbies. And that allows me to do things that are really aggressively useless, but that better reflect my personal heart.
One of the things I did was teach Nintendo games how to play themselves. Not really. But teach computers how to play Nintendo games. This was in 2013, before DeepMind and all of those things. This learn fun and play fun, that thing learns to play Mario pretty well, and Tetris very badly, although it figures out some tricky ways of getting around the fact that it's so bad at it.
I once alphabetized Star Wars, not for any good reason. And I've recently made this compiler that produces 16-bit DOS executables, EXE files. They run on your DOS computer that you still have, probably. Because it's only printable x86 opcodes, things you could type on your keyboard, including the header and so on, with no self-modifying code. So that's weird.
I once figured out how to sort of semi-automatically turn Nintendo games into 3D. And so OK, yes, so I know what you're thinking. This guy is a total one-trick pony. He will take some, list something nostalgic-- media or system-- and then bring it into the modern era, throw some computation at it. And it's got these pandering references to our youths in it.
And this is true. And we're going to see another example of that. We're going to see another example of that today.
But since it's Deconstruct, I thought I could also sort of belabor the underlying joke by layering meta layers on top of it. And that's part of what makes this talk so weird. OK. So bear with that.
One other thing to say. I forgot what the other thing was. Oh, yeah, right. So humor is a thing that pervades my work, hopefully. And I think one could say this is useless. What is the point of that?
But I think it's actually really useful to do humorous work, work that strikes your fancy, because it will sometimes lead to really interesting feelings. One of those is the elation of getting a joke. And when you learn a technical subject, and you're able to get a deeply technical joke, I think that's a really fun feeling. And it's a super power that's created by your technical understanding.
But also sometimes you just happen upon neat ideas that could enhance the way you think about other problems. So even though this is an aggressively useless project, I hope that some of the concepts, as I belabor the meta joke, I hope you think about them some time in the future. And I hope that the talk is memorable.
Now I mentioned I'm doing a project. And that was a really tricky, hard one for me. And I almost gave up on it many times. And since I have this speaking engagement, I was going to do a backup talk, which was about these sort of jokes in computer science. So it was going to look like that. [INAUDIBLE] I didn't need to do that, because I succeeded.
By joke scheme is I don't mean the DTD, although that is a good joke schema. I mean what is funny. And this is the kind of problem that some really boring, pretentious philosophers have studied to death, and desiccated all of the fun out of humor, and turning it into a kind of prune dust to put in books. But there's something we can learn from that. And since I'm doing this sort of meta-meta thing, let's talk a little bit about it.
And I recognize that we're right before lunch. These are food-themed jokes. So also, if you have sensitivity to that, be careful.
Here's an example of something that's not funny. I think this was WC Fields, or some old-timey comedian, who said, show me a movie of a man eating a boot who has a look of disgust on his face, and he's sweating, and so on. And that's not funny. But funny is if you show-- I know, the rapturous laughter-- you show me a movie of a distinguished person with a neutral look on his face eating a boot. And that's funny.
I'm not sure either of those things are funny, personally. I think it's maybe meta funny that boots were once considered the nexus of slapstick comedy, eating them. But there is something to this, which is the idea of a violation of expectations.
Of course, if I'm eating a boot and-- oh, that's kind of bad. Well, we'll see if that keeps up. If I'm eating a boot, and I'm not thinking anything of it, then that's a violation of our expectations.
Let's do another funny-not-funny. Remember to laugh on the right slides. Mario-- no. Not funny. Are you really laughing to the instructions? That's OK.
Mario eating a boot. Not funny. Here again with the boots. But here the joke is retro. Oh, I showed you Mario. And you remember that from your youth. Ha-ha, right? I think that's not a good example of a joke.
But in contrast, it might be funny for Samus from Metroid to eat Tide PODS, at least in the year 1986, because Tide PODS didn't exist in 1986, right? So this, I'm contrasting here retro Mario getting a boot-- boots existed in 1983, or whatever-- with anachronism, a violation of that expectation.
Another example. I actually thought at first self-reference, that's the best computer science joke, other than abstraction. But actually I don't think this is true. If I just say hello, my name is Tom 7, that is not funny anymore.
And yet in contrast, I think this idea of a strange loop is actually really funny. That might be the best joke in computer science. I'll give you an example of a famous one. This is a quine.
So if I take that source code at the top, which I've rudely broken into multiple lines so that it's no longer a proper quine, but imagine it on one line. And I pass that to GCC, and I compile it to quine.exe, because I like to use Windows-- I'm an old guy-- and run it, then it produces as output the same source code that created it.
This is awesome. And this predates me. I certainly didn't invent this. I wish I had. But it's such a lovely little gem in computing.
It's a loop. It's a strange loop. And I think this is maybe the canonical example of a strange loop.
But I contend that that last step is actually the one that is the most interesting here. And why do I say that? Well, there's this idea that as the executable that's produced by source code, the source code is a precursor to the executable, to the binary, right? It comes before it.
And yet here we have a binary that produces source code. That's in the wrong order. That's a violation of some kind of inherent expectations.
So enter this term "improper hierarchy," which describes this joke schema in computer science. And if you haven't heard of improper hierarchy, don't worry. You're not alone. I had also never heard of it, and so I made it up. Yeah, you can just do that. No one will even stop you.
So we're going to do a joke about improper hierarchy just as an example here. So it's like you have some implied hierarchy or explicit hierarchy, right? But you're going the wrong direction in it.
So here, anachronism is an example, because information, like Tide PODS, pure information, flow to the past from the present or future. And, just to belabor it a little bit, you've got self-reference. You've got strange loops, which are a subset of that. But improper hierarchy, which I drew to be this huge expanse, is a generalization of strange loops that don't necessarily have to be loops. Anachronism doesn't need time travel loops, or paradoxes.
And here's a self-promotion tip for you. Manage to get yourself on a slide with Descartes and Douglas Hofstadter. Draw the specious parallels. No one will stop you, as far as I know.
In order to do that, though, we need to talk about the Nintendo motherboard, the NES-- Nintendo Entertainment System-- from 1983ish, depending on what country you live in. This is the motherboard. I drew it for you. We've got a little bit of a low-resolution display here.
And it's got a CPU. It's got a PPU. Like modern computers, there's this sort of bicameral mind of the computational subsystem and the graphics subsystem. They called it pictures back in those days, because I guess it was a little bit old-timey. And so the picture-processing unit processes pictures. The central processing unit processes centers.
And then there's this asshole. This is, I don't know what it stands for, some copyright something copyright. This thing's only job is to ruin your day by resetting the Nintendo if it detects that something is amiss related to copyright.
And so you know the trope about blowing in cartridges. Don't do that. It does nothing. And actually the only thing that can make your Nintendo reboot and blink its light is this stupid chip that Nintendo put there on purpose to ruin your day. So thanks a lot Nintendo.
It's also an analog device, really. I mean, it's digital. But digital isn't real. It's an abstraction. So there's these other components on the board, like these capacitors, which I think of as goombahs. That's the only point of saying that. I just like that.
Right. The CPU and the PPU kind of talk along these lanes. They don't talk to each other very much, although it is possible to do that a little bit. It's slow to do that. They're actually mostly just operating independently.
Those connect down to game. "Game pack" is the formal term for this. This one is called Footballs. And inside that, if you open it up, as I did when I was 7, you find this hilariously small circuit board. You're like, I paid 60 1980s dollars for this?
And the pins follow. It's a obviously physical thing here, where they connect up to the sort of graphic side and the CPU side to the Nintendo. Does that makes sense? So that's the CPU ROM on the left that contains the program. That's the PPU ROM on the right that contains the graphics. And there's a corresponding CIC to collaborate with its friend to not reset your computer.
But you look at this, and you know, in Japan, the cartridges are the size of that circuit board. They're just the right size. And when we imported this to United States, we were like, USA number one! And we made it four times bigger than it needs to be, like the SUV of game packs, which is going to come in handy today.
So this gives me an idea, which is part of the next phase of the talk. So here's the normal set-up. You have the Nintendo, and you have your cartridge.
And you know of emulation. Emulation is where I take the Nintendo, chuck that out, because it's blinking its red light. And I replace it with a software implementation of the Nintendo. Then I can take a cartridge and dump it to a file. And I can play Nintendo games on my computer, which is great. Wish I'd invented that, too.
But I maybe invented the following thing, which is reverse emulation. We're going to flip that idea on its head.
So here I'm going to take the Nintendo. And it's going to be an unmodified Nintendo. Don't change anything about that. But I'm going to take the cartridge, and I'm going to chuck that in the trash-- except for the CIC chip. I need the CIC chip.
I'm going to replace its guts. It's basically these ROMs, which are the most boring electrical component-- well, digital electrical component. I'm going to replace the ROMs with ridiculously active circuitry, which is hardware and software, which will pretend to be ROMs, but isn't. And this is going to enable me to unlock abilities that the Nintendo always had, but that it could not use, because it was using dumb ROMs instead of cool reverse-emulated stuff. Are you excited about that?
And in a case it's not clear, this talk is taking place on an unmodified Nintendo with a cartridge that I made that fits right in there, and is plugged through this thing in a ridiculously janky way. And that's the reason for-- oh, good.
[APPLAUSE]
I'm glad you appreciate that. That was a lot of work. And that's the reason for the flashing, which we'll get to why that's really happening in a bit.
So here, let's talk about reverse simulation. This is the Raspberry Pi Zero W, which is a tiny computer that ostensibly costs $10, which is amazing. You can only buy it one at a time. So I think it's kind of a trick, like it's Lost Leader.
But still, $10, they'll really send it to you. And it's got a gigabyte of RAM-- sorry, 1 gigahertz CPU and 256 megs of RAM. Who cares? Specs. It's got wireless. Pretty amazing little computer that you can kind of give away as party favors, it's so cheap.
And here's my fantasy schematic. What I'm going to do is I'm going to take the cartridge header, which plugs into the Nintendo. And that's super slow, like it's a joke. It's like a megahertz.
And on the other side I've got my Raspberry Pi, which is infinitely fast. And I'll just connect them up. It'll send me addresses, and I'll send it data. And all will be well.
I'm used to this being possible. I work on emulation, too. And my emulator has been working on it. I think it can run 28,000 times faster than a Nintendo on my desktop. And it can fit something like 32 million instances of a Nintendo game in RAM at once. So that's orders of magnitude.
We'll see. I was not totally naive. I'm like, OK, I'll probably need a resistor in there, in the schematic.
[LAUGHTER]
Usually need those. But it turns out to be more complicated than that.
But there was hope. I was given hope when I peered at the vulnerable underbelly of the Nintendo. And this is the Nintendo you're seeing on stage, a photograph of it, which I put inside the Nintendo on a cartridge.
And if I zoom in on that sticker, I'll see, oh, this is FCC compliant, blah, blah, blah. This device may not cause harmful interference. And 2, this device must accept any interference that may be received, including interference that may cause undesired operation.
I take that to mean, also, desired operation. So this behavior is required by law. So that was giving me some hope.
Let's talk briefly about how it works. I know a lot about this topic. And if you want to talk over drinks, please. But I've simplified it here so that we can get through the important stuff.
Basically the Nintendo's a graphics subsystem. And that's the main thing that I'm going to reverse emulate, is an array of characters. You could just imagine an ASCII matrix on textmode on your computer.
And then the graphics that are those characters, like the pixels that make up the letter A. Those graphics don't necessarily need to be letters. In fact, they're usually stuff like question mark box, but also letters. And those are 8x8 pixels. And they're 2-bit color.
So this is what a screen of Mario looks like. Those are the 8x8 blocks. And here it is in sort of sepia tone 2-bit color.
I can also apply colors. I've got 13 colors to choose from. And I can change which ones those are, although that's a little involved. But I can select regions of the screen to use one of those four palettes for their 2-bit color. And 2-bit color yields four colors, right?
So this is what this looks like. Oh, I always mess this slide up. These are all of the colors that a Nintendo can display. But I can't even show them all on the screen at once, because I can only get 13 colors in the palette. So I'm strobing through them.
This is what that looks like in Mario. The color regions are those larger blocks. And this is the reason why graphics appear to not be those 8x8 things, but those 16x16 things, like question mark box, is you can only change the color resolution of the Nintendo that often.
But this also yields some great observations. I love this one. These things here, the cloud and the bush, are actually the same exact tile graphics with different colors applied to them. That's a mind-blowing, brilliant move to me. I really love that.
So I talked about color resolution. That's an apparent limitation in the Nintendo. But it's something that, by reverse simulating it, by flipping it on its head, I can actually get around that a little bit. So normally, 16x16 blocks often have the same colors.
But here's how the rendering actually works. You don't need to read this. It's actually rendering an 8x1 strips as the hardware is going through. And it will read the color bits-- the color attributes-- multiple times-- I guess 16 times per block that it renders. It reads that same color bits.
But if I'm actively choosing what to return for any given address, I can return inconsistent results every time it reads the ROM, or RAM, for a particular color. Which means I can change the color very often. I can change it for every one of these 8x8 strips, instead of at this large granularity. That's the kind of thing I mean when I say unlocking the abilities that the Nintendo does have, but sort of on paper, if you read the development manual, it doesn't.
Another limitation is the update speed. So here's how the Nintendo's normally rendering. Its NTSC scanning, 240 scan lines on your good old CRT TV, or this extremely fancy thing.
And then there's 20 VBLANK cycles at the end, where it's waiting for the CRT to refresh. And that end, which is only 1/13 of the time. This is more or less to time scale.
And that end is the only time that you can actually write to the PPU. So you can actually change it, for example, the palette. Or you can change that matrix that makes up the map.
So that's really a bummer, because in Mario, this is the amount of the screen that can change per frame. You can't repaint the entire screen by any stretch. If Mario moves too fast, the screen wouldn't be able to keep up, because you could only replace about two columns' worth of the screen.
Yeah, so I'm going to be able to get around that, because I can, of course, replace all of the bytes anytime they're read, including during the same scanline. So I should be able to do full-screen animations. You've even seen some of that already, now that I've spoiled, that this talk is using the very technology it describes.
Let's get a little bit down into some of the harder problems I needed to solve to do this, just because we're at a technical conference. But I'm going to try not to spend too much time on this, because I know I'm running-- well, there's a lot of material.
So here's an oscilloscope image of this Nintendo communicating with a cartridge in digital mode. And another weird thing I'm doing here is I've taken a reading of these traces. I've turned that into a computer image, then put it back into the Nintendo. And then those are flowing over those very same traces in order to render an image of the oscilloscope.
That's, I don't know. That's kind of a weird loop to me. Seems a little wrong.
And the timings here are actually pretty hard, even though it's supposedly kind of a slow thing. We're talking about nanoseconds. And so basically you have this read and write clock, or wait and write clock. And then you have addresses that come in, and data that comes out.
So first test is how fast can the Raspberry Pi do this same kind of thing? And the answer is that fast. And that is not really a lot of headroom. I'm used to orders of magnitude. Here it can do about three times faster than the Nintendo wants.
And worse, if I do the normal thing, which is when the read cycle begins, I read the address, and then write the data that corresponds to that, and I hand optimize that assembly code, and I turn off memory barriers like a cowboy, it's still late. It does not happen in time, and it completely doesn't work. It's really hard to debug, too, it turns out.
So here's an example of a kind of trick you do, which is a good life lesson. If you can't submit on time for the conference, submit on time for next year's conference. You'll be super early.
So what I do is as soon as the wait cycle begins, I know something is going to happen, and I write my data. And it takes 200 nanoseconds to get out onto the bus. And then I can read the addresses. And this gets me right there on time, with the major downside that my write happens before my read. So my write can't depend on the address that was read. So that's a little bit hard.
So what I need to do is I'm basically trying to keep up with the PPU, and predict where it's going on the screen, and then have the right byte cued up. And then I adjust my guess of where I am based on the read that comes in. And this is the main reason. This timing is the main reason why you're getting all that jank on the screen.
Of course, it's not really a digital device. This is a fantasy. This really looks more like that, which is itself another fantasy. It's an analog device. And I can't tell you how wrong my fantasy schematic is.
All you EE geniuses out there are really, I'm sure, laughing it up inside. But if you do this, you'll actually break the Nintendo, probably, and your Raspberry Pi. One of them is 3.3-volt CMOS. One of them is 5-volt CTL.
So you're like, oh, I know this. I'm going to use a resistor, uh-huh, and a transistor. And I'll put a 0 and a 1. But that doesn't work. It's totally, totally wrong. I was not knowledgeable about electronics, I guess. Because it's a shared bus. And, in fact, addresses and data use the same bus.
So you learn about tri-state buffers, and how this stuff actually works, which is cool, and I could explain to you now. I think I understand it. But that turns out that, oh my goodness, even though I built this thing out of transistors, it still takes 180 nanoseconds to discharge.
How is that possible? Isn't everything just made of transistors? Aren't they the fastest thing in the world?
Well, it turns out in the '60s they invented a way of preventing these things from saturating too much. And that's called the Schottky transistor. And there's this whole line of successive improvements to TTL. And if you look in here, it's original low-power, high-speed, Schottky, low-power, advanced. And then the skinless low-power Schottky is probably what we're going to invent next. That was a Engelbart joke.
But right about the right era for the Nintendo you'll see, oh, low-power Schottky, which has about the right timing characteristics that you're measuring. And then you're like, oh, wait. I'm looking at the Nintendo motherboard, and here's a chip on it-- SN74ls. These letters mean something?
And so you think, oh, I know what I'll do. I'll start copying off the Nintendo design. So I'll put some of those goombahs on my board as well. But, OK, so all you EE geniuses smugly out there, what is this component that you're copying off a-- you know it's a trick, now. You're copying this off a Nintendo circuit board.
It looks exactly like a resistor. It's got the stripes. It's exactly a resistor package. But it turns out it's a capacitor. In Japan in the '70s they would do this because they had machines that could place-- pick a place-- resistors, but not capacitors. So you can easily be led astray also by studying-- I know-- by studying those that came before us.
So I built this thing. This is the more the real-world schematic. Actually, it's the Gerber files. It's kind of complicated. I called it that PPU. So it's got a good logo. But actually I made a mistake in naming this, because I don't know how to pronounce words that start with two P's. It's like maybe P-P-Puppy.
And this is what it looks inside the actual cartridge. That's a picture of this actual cartridge that I've put inside the cartridge to display through the Nintendo onto the screen, the picture of the thing. You can see the dog logo on that.
It's got one of these guys on it. And this is a real slap in the face, because even though it's a stupid little copyright chip, you've got to harvest these from real games. So every time I want to make one of these, I've got to buy something on eBay, and take it apart, and desolder it, which is a pain. Often break it.
And Nintendo games are getting kind of expensive now. The cheapest Nintendo games to harvest for their components are more expensive than the cheapest Super Nintendo games, which I consider to be, I guess, I consider that to be wrong.
So there's also this big chip that's the ROM for the CPU. I mostly am doing this for the graphics subsystem, but there has to be a program that does something. And I'm going to tell you about that. And then we're going to get onto a demo-- another demo.
So the CPU's job is going to be to update the palette-- the global palette-- for the screen. And it's the only thing that can do that. The palette isn't read out of the same memory that the graphics are read out of. So it has to be done by the CPU.
And it has to do the input. So I'm controlling this, because I didn't want to rely on those clickers. Those are having a lot of technical issues, don't you think? This is really solid in comparison.
So here's basically how that works. Normally the screen is rendering scan lines, and it's the PPU talking to my board. Just that. CPU is just sitting there waiting for something to happen.
Then an interrupt happens. Interrupt tells CPU, I'm in VBLANK. You can now communicate with the PPU, which the PPU is where I'm going to store the palette.
But I also need to get what the next palette is from my board, which tells it that thing, I should have a red and blue and a green on this screen. So I need to communicate it with it. But the only thing I can do is read bytes out of it.
So I do this trick called knocking, which is I'm going to read in a specific sequence to tell the board, hey, I'm not rendering now. I'm trying to sync up with you. I'm the CPU.
So what I do is I just basically read. And remember, this is annoying, because the PPU-- sorry. My board is sort of predicting what's happening. It's not actually responding to these in real time. So it takes some steps.
But I read in descending memory locations, which should never happen, since the PPU renders left to right. I do that until I kind of sync up with it. And once I'm synced up, now it can stream data to the CPU.
And what it does is sends what the palette should be for the next screen. And it also sends the scroll position, because I'm going to do some tricks later with modulating the scroll position to improve the graphics. Improve the graphics.
So that's great. I can send the palette. By doing this trick of knowing that behind the scenes there is something that's smart, it has hysteresis about the order of reads.
So mission accomplished on that. But I also want to be able to report inputs. The CPU can read the controller. So I want to send data to the board.
Turns out this can be done the same way. So once I'm synced up, the addresses, you normally think of a read as a one-way operation. You give the data to me. But if I encode in the addresses that I read some information, then I can pass it to you, and you can send me data that you don't care about, if you like.
So basically just jam the joystick bits. There's 8 bits per controller. 8 bits. Makes sense. Inside those reads. So that's how that works.
Good. Well, bad actually. Because I used up basically an entire VBLANK doing this. I have 2/3 of it devoted to that process, which gives me 514 microseconds to do interesting stuff on the Raspberry Pi, which is not really enough. It's enough to do this slide presentation. But I wanted to do more interesting things to get my joke done.
So we are going to switch-- oh, I'll show you the slide that tells me to switch. We're going to switch to another cartridge. This cartridge, I've got it in an anti-static bag, because this one is really, I don't know if you can see. But it's like it didn't quite fit in there. So I thought this was not quite as faithful to the dream. But it is, I think, a pretty good joke.
So if it boots, which is definitely risky. Also vote for me for riskiest talk. Weirdest and riskiest.
Right. Unfortunately, I was using up the entire CPU. This is good. This is a good sign. I used up the entire CPU.
But I want to be able to do something else interesting. So here's a multicore Raspberry Pi. It's the highest end Raspberry Pi. That's why it doesn't fit. And also, I needed to upgrade the power supply. And it gets kind of hot, so we can't-- I don't think we can safely run this for that long.
[LAUGHTER]
And it might be F-sucking right now, FS checking. Well, hopefully it will work. Anyway, I'll tell you about what it is. And then maybe we'll get to see it.
So it's a multi-core system. And the reason I didn't just do the talk off this was that I was unable to get it to-- so in the talk version, the talk cartridge-- oh, good. Well it's got the wrong background color. But that should be OK.
In the talk version, I basically disable Linux. So when I'm running, I tell the low-level hardware to turn off the timer interrupts. And then Linux can't ever come back, which is really good, because then I can be hitting these micro-second or nanosecond deadlines.
On this one, unfortunately, because I have two cores, I couldn't figure out a way to turn that off without disabling one of the cores. And so Linux is basically waking us up every second, be like, hey, do you want to schedule anything? I was like, I was doing something. Every second, like that. So that's what's causing the blinking.
[LAUGHTER]
But I hope you'll think it's worth it. So I'll press start.
And then here is my joke. I don't know why the background is pink.
But what we have here is the launch title for the Super Nintendo. It's the 16-bit successor of the Nintendo, running-- oh, I'm glad you appreciate it-- running inside the Nintendo on a cartridge that I made. But this is an unmodified Nintendo. It should not be able to make 16-bit graphics and so on, at least at this level of fidelity. So I can play this.
One weird thing is that the controller buttons, I found this out. So I don't have as many buttons as the Super Nintendo has. But they actually line up. Up is up. Up on the Nintendo is up on the Super Nintendo by some, I don't know, coincidence. Or maybe there's just some laws about that. And the FCC of Japan. Man, I really wish the background wasn't pink.
So what else to say about this? Yes. So I'm doing a bunch of tricks. Actually the hardest part about this is not-- oh, there's a lot of latency. Ooh-ooh. This may not be a 100-- oh.
[LAUGHTER]
I was going to say, this is probably not going to be 100% speed run. I'm going to give it one more shot while I talk.
So one of the tricks I'm doing is, the Nintendo only has those 13 colors. So I am switching the palette and the scroll offset a lot in order that every other frame-- I need this for safety-- in order that every other frame-- oh, my goodness. Yeah, you can't tell, but there's maybe 500 milliseconds of latency between my presses. It's not normally like that. It's just this cheap '80s system that this opera house has.
[LAUGHTER]
I'm just kidding. I'm just kidding. I love you, AV folks. Everything is working really great.
Yeah, so I'm vibrating the screen back and forth at 60 Hertz so that I don't get the same sort of 8x1 grid that the Nintendo sort of forces upon me. And this gives me smoother both color resolution and sort of pixel resolution.
So OK, that's that. I hope you understand this, this, the improper hierarchy, how this leads in. I considered doing a quine, a quine Nintendo. And that would look like this. This is where I'm running a Nintendo emulator inside the Raspberry Pi, and then processing all its graphics.
Oh, you like that one better. Ah. I actually thought that the Super Nintendo one was a better joke when I was thinking about it. But now I know. The clapometer never lies.
I want to play, but I know it's boring for you to watch that. So, oh, my, the graphics have really gone awry. Well, let's just get back in the talk. I'm almost done. This is the very end.
So I could reverse emulate the CPU. And you could unlock some better. You could make the CPU faster, basically, by streaming it, only constant reads and writes. But I want to just connect this back, just deconstruct. So we got to have some kind of connecting it back to big ideas.
So this technology is good, perhaps, for nostalgic hardware, making nostalgic hardware, bring it into the modern era-- calculators, the Nintendo, Super Nintendo, Nintendo 64, whatever-- the Game Cube.
But maybe there's some other uses. What's another piece of hardware that we have nostalgia for? How about the human brain?
I feel like--
[LAUGHTER]
I feel like we should think about these things, because AI, it seems obvious to me that we will succeed at AI. And that's maybe not a good thing. I worry about that-- not only because it could destroy us, and all of society, but I also I really don't the idea of feeling left behind.
So we have AIs today that can play chess so well that it's not even fun to play at them, just as one kind of boring example. And I don't want to live in a world where there's these ultra smart, really cool robots, that are like, oh, you invented reverse emulation. Look what I invented. And then they show me this way better stuff.
So I would like it if we can somehow figure out how to make use of our legacy hardware in a future world. And one of the things I like about this thing, this talk, which is mostly a joke, is it taught me a little bit about what it might feel like to do that.
So I'm not talking about uploading your brain into the cloud or something. That would be emulation. I'm talking about reverse emulation.
So what if I replaced part of my legacy hardware, like my memories, with something that was vastly better than the actual hardware that I have? And obviously I haven't solved that by any stretch. And I'm not even working on it, because you can see how bad I am at electronics.
But if we were to solve that, if we understood the sort of logic family that is the human mind, and we could connect that up with, I would probably splurge for the high-end Raspberry Pi, because it's like my brain. Maybe I could have computer super powers.
And maybe that would feel like the following. I'm trying to understand what are the prime factors of this number, right? It's a big, boring number. Computers, this is easy for them. For us, it's a hard problem; sort of intractable problem.
So maybe I might be able to make it feel like I just know all of these things. Everything is just a part of my rote understanding of facts.
So I look at that number, and then I think, oh, I'll just sing myself the ABCs. (SINGING) A-B-C-D.
But then this board that I've embedded in my brain picks up. It notices this knocking. And it thinks, oh, I'll just now replace his memory with the answer to his question. (SINGING) A-B-C-D 472 866 02.
We'd also have to figure out how to make it rhyme, otherwise it wouldn't be very satisfying. But I think that, I don't know, it might work. And that tickled my fancy when I was working on this thing.
So that's a good thing, except we have a problem, which is that it is outlawed. The galaxy federal police in M510-- the year M510 made an order to destroy the Metroid of planet Symbath, and the mother brain, mechanical life brain. So we're also going to need to pass some new laws in order to do this. But otherwise I think we're in pretty good shape.
That's my talk. Visit me online at Tom 7, wherever that is-- Twitter or Google or something. And yeah, let's enjoy lunch. Thank you.
[APPLAUSE]
Transcript
(Editor's note: transcripts don't do talks justice. This transcript is useful for searching and reference, but we recommend watching the video rather than reading the transcript alone! For a reader of typical speed, reading this will take 15% less time than watching the video, but you'll miss out on body language and the speaker's slides!)
Our next speaker, who is definitely not on stage right now, please welcome Tom.[APPLAUSE]
Hi. I'm Tom 7. Someone told me it was possible to rise out of the stage, since we're in an actual opera house. So at least I got to kind of mimic that. But maybe next year.
So here's my talk. It is a weird talk, so I'm trying to kind of go for, when we vote at the end on weirdest talk, and most confusing talk, vote Tom 7.
And along those lines, I have to do this. Not really a joke. There's some flashing in this talk. It's sort of part of the nature of the work.
And so if you're sensitive to that, this talk may not be for you, or you may need to close your eyes or something. It gets particularly bad in the coda. So I'll try to warn you when that's happening. This slide is gratuitous. But some of the other stuff that's going on there is just part of what's going on.
But speaking of what's going on, who is this man in front of you? Who is Tom 7, if anything? I have done some things you may have seen. And I'm just going to show you these, A, because maybe it makes you think, oh, that guy; I'll listen to this talk. But B, because I think it's characteristic of the kind of work that I do, and my talk is a meta-meta joke about a project or something. So that's part of why it's so confusing.
So this helps you understand a little bit about where I'm coming from. By the way, I do have a normal day job where I contribute to society. So these are my hobbies. And that allows me to do things that are really aggressively useless, but that better reflect my personal heart.
One of the things I did was teach Nintendo games how to play themselves. Not really. But teach computers how to play Nintendo games. This was in 2013, before DeepMind and all of those things. This learn fun and play fun, that thing learns to play Mario pretty well, and Tetris very badly, although it figures out some tricky ways of getting around the fact that it's so bad at it.
I once alphabetized Star Wars, not for any good reason. And I've recently made this compiler that produces 16-bit DOS executables, EXE files. They run on your DOS computer that you still have, probably. Because it's only printable x86 opcodes, things you could type on your keyboard, including the header and so on, with no self-modifying code. So that's weird.
I once figured out how to sort of semi-automatically turn Nintendo games into 3D. And so OK, yes, so I know what you're thinking. This guy is a total one-trick pony. He will take some, list something nostalgic-- media or system-- and then bring it into the modern era, throw some computation at it. And it's got these pandering references to our youths in it.
And this is true. And we're going to see another example of that. We're going to see another example of that today.
But since it's Deconstruct, I thought I could also sort of belabor the underlying joke by layering meta layers on top of it. And that's part of what makes this talk so weird. OK. So bear with that.
One other thing to say. I forgot what the other thing was. Oh, yeah, right. So humor is a thing that pervades my work, hopefully. And I think one could say this is useless. What is the point of that?
But I think it's actually really useful to do humorous work, work that strikes your fancy, because it will sometimes lead to really interesting feelings. One of those is the elation of getting a joke. And when you learn a technical subject, and you're able to get a deeply technical joke, I think that's a really fun feeling. And it's a super power that's created by your technical understanding.
But also sometimes you just happen upon neat ideas that could enhance the way you think about other problems. So even though this is an aggressively useless project, I hope that some of the concepts, as I belabor the meta joke, I hope you think about them some time in the future. And I hope that the talk is memorable.
Now I mentioned I'm doing a project. And that was a really tricky, hard one for me. And I almost gave up on it many times. And since I have this speaking engagement, I was going to do a backup talk, which was about these sort of jokes in computer science. So it was going to look like that. [INAUDIBLE] I didn't need to do that, because I succeeded.
By joke scheme is I don't mean the DTD, although that is a good joke schema. I mean what is funny. And this is the kind of problem that some really boring, pretentious philosophers have studied to death, and desiccated all of the fun out of humor, and turning it into a kind of prune dust to put in books. But there's something we can learn from that. And since I'm doing this sort of meta-meta thing, let's talk a little bit about it.
And I recognize that we're right before lunch. These are food-themed jokes. So also, if you have sensitivity to that, be careful.
Here's an example of something that's not funny. I think this was WC Fields, or some old-timey comedian, who said, show me a movie of a man eating a boot who has a look of disgust on his face, and he's sweating, and so on. And that's not funny. But funny is if you show-- I know, the rapturous laughter-- you show me a movie of a distinguished person with a neutral look on his face eating a boot. And that's funny.
I'm not sure either of those things are funny, personally. I think it's maybe meta funny that boots were once considered the nexus of slapstick comedy, eating them. But there is something to this, which is the idea of a violation of expectations.
Of course, if I'm eating a boot and-- oh, that's kind of bad. Well, we'll see if that keeps up. If I'm eating a boot, and I'm not thinking anything of it, then that's a violation of our expectations.
Let's do another funny-not-funny. Remember to laugh on the right slides. Mario-- no. Not funny. Are you really laughing to the instructions? That's OK.
Mario eating a boot. Not funny. Here again with the boots. But here the joke is retro. Oh, I showed you Mario. And you remember that from your youth. Ha-ha, right? I think that's not a good example of a joke.
But in contrast, it might be funny for Samus from Metroid to eat Tide PODS, at least in the year 1986, because Tide PODS didn't exist in 1986, right? So this, I'm contrasting here retro Mario getting a boot-- boots existed in 1983, or whatever-- with anachronism, a violation of that expectation.
Another example. I actually thought at first self-reference, that's the best computer science joke, other than abstraction. But actually I don't think this is true. If I just say hello, my name is Tom 7, that is not funny anymore.
And yet in contrast, I think this idea of a strange loop is actually really funny. That might be the best joke in computer science. I'll give you an example of a famous one. This is a quine.
So if I take that source code at the top, which I've rudely broken into multiple lines so that it's no longer a proper quine, but imagine it on one line. And I pass that to GCC, and I compile it to quine.exe, because I like to use Windows-- I'm an old guy-- and run it, then it produces as output the same source code that created it.
This is awesome. And this predates me. I certainly didn't invent this. I wish I had. But it's such a lovely little gem in computing.
It's a loop. It's a strange loop. And I think this is maybe the canonical example of a strange loop.
But I contend that that last step is actually the one that is the most interesting here. And why do I say that? Well, there's this idea that as the executable that's produced by source code, the source code is a precursor to the executable, to the binary, right? It comes before it.
And yet here we have a binary that produces source code. That's in the wrong order. That's a violation of some kind of inherent expectations.
So enter this term "improper hierarchy," which describes this joke schema in computer science. And if you haven't heard of improper hierarchy, don't worry. You're not alone. I had also never heard of it, and so I made it up. Yeah, you can just do that. No one will even stop you.
So we're going to do a joke about improper hierarchy just as an example here. So it's like you have some implied hierarchy or explicit hierarchy, right? But you're going the wrong direction in it.
So here, anachronism is an example, because information, like Tide PODS, pure information, flow to the past from the present or future. And, just to belabor it a little bit, you've got self-reference. You've got strange loops, which are a subset of that. But improper hierarchy, which I drew to be this huge expanse, is a generalization of strange loops that don't necessarily have to be loops. Anachronism doesn't need time travel loops, or paradoxes.
And here's a self-promotion tip for you. Manage to get yourself on a slide with Descartes and Douglas Hofstadter. Draw the specious parallels. No one will stop you, as far as I know.
In order to do that, though, we need to talk about the Nintendo motherboard, the NES-- Nintendo Entertainment System-- from 1983ish, depending on what country you live in. This is the motherboard. I drew it for you. We've got a little bit of a low-resolution display here.
And it's got a CPU. It's got a PPU. Like modern computers, there's this sort of bicameral mind of the computational subsystem and the graphics subsystem. They called it pictures back in those days, because I guess it was a little bit old-timey. And so the picture-processing unit processes pictures. The central processing unit processes centers.
And then there's this asshole. This is, I don't know what it stands for, some copyright something copyright. This thing's only job is to ruin your day by resetting the Nintendo if it detects that something is amiss related to copyright.
And so you know the trope about blowing in cartridges. Don't do that. It does nothing. And actually the only thing that can make your Nintendo reboot and blink its light is this stupid chip that Nintendo put there on purpose to ruin your day. So thanks a lot Nintendo.
It's also an analog device, really. I mean, it's digital. But digital isn't real. It's an abstraction. So there's these other components on the board, like these capacitors, which I think of as goombahs. That's the only point of saying that. I just like that.
Right. The CPU and the PPU kind of talk along these lanes. They don't talk to each other very much, although it is possible to do that a little bit. It's slow to do that. They're actually mostly just operating independently.
Those connect down to game. "Game pack" is the formal term for this. This one is called Footballs. And inside that, if you open it up, as I did when I was 7, you find this hilariously small circuit board. You're like, I paid 60 1980s dollars for this?
And the pins follow. It's a obviously physical thing here, where they connect up to the sort of graphic side and the CPU side to the Nintendo. Does that makes sense? So that's the CPU ROM on the left that contains the program. That's the PPU ROM on the right that contains the graphics. And there's a corresponding CIC to collaborate with its friend to not reset your computer.
But you look at this, and you know, in Japan, the cartridges are the size of that circuit board. They're just the right size. And when we imported this to United States, we were like, USA number one! And we made it four times bigger than it needs to be, like the SUV of game packs, which is going to come in handy today.
So this gives me an idea, which is part of the next phase of the talk. So here's the normal set-up. You have the Nintendo, and you have your cartridge.
And you know of emulation. Emulation is where I take the Nintendo, chuck that out, because it's blinking its red light. And I replace it with a software implementation of the Nintendo. Then I can take a cartridge and dump it to a file. And I can play Nintendo games on my computer, which is great. Wish I'd invented that, too.
But I maybe invented the following thing, which is reverse emulation. We're going to flip that idea on its head.
So here I'm going to take the Nintendo. And it's going to be an unmodified Nintendo. Don't change anything about that. But I'm going to take the cartridge, and I'm going to chuck that in the trash-- except for the CIC chip. I need the CIC chip.
I'm going to replace its guts. It's basically these ROMs, which are the most boring electrical component-- well, digital electrical component. I'm going to replace the ROMs with ridiculously active circuitry, which is hardware and software, which will pretend to be ROMs, but isn't. And this is going to enable me to unlock abilities that the Nintendo always had, but that it could not use, because it was using dumb ROMs instead of cool reverse-emulated stuff. Are you excited about that?
And in a case it's not clear, this talk is taking place on an unmodified Nintendo with a cartridge that I made that fits right in there, and is plugged through this thing in a ridiculously janky way. And that's the reason for-- oh, good.
[APPLAUSE]
I'm glad you appreciate that. That was a lot of work. And that's the reason for the flashing, which we'll get to why that's really happening in a bit.
So here, let's talk about reverse simulation. This is the Raspberry Pi Zero W, which is a tiny computer that ostensibly costs $10, which is amazing. You can only buy it one at a time. So I think it's kind of a trick, like it's Lost Leader.
But still, $10, they'll really send it to you. And it's got a gigabyte of RAM-- sorry, 1 gigahertz CPU and 256 megs of RAM. Who cares? Specs. It's got wireless. Pretty amazing little computer that you can kind of give away as party favors, it's so cheap.
And here's my fantasy schematic. What I'm going to do is I'm going to take the cartridge header, which plugs into the Nintendo. And that's super slow, like it's a joke. It's like a megahertz.
And on the other side I've got my Raspberry Pi, which is infinitely fast. And I'll just connect them up. It'll send me addresses, and I'll send it data. And all will be well.
I'm used to this being possible. I work on emulation, too. And my emulator has been working on it. I think it can run 28,000 times faster than a Nintendo on my desktop. And it can fit something like 32 million instances of a Nintendo game in RAM at once. So that's orders of magnitude.
We'll see. I was not totally naive. I'm like, OK, I'll probably need a resistor in there, in the schematic.
[LAUGHTER]
Usually need those. But it turns out to be more complicated than that.
But there was hope. I was given hope when I peered at the vulnerable underbelly of the Nintendo. And this is the Nintendo you're seeing on stage, a photograph of it, which I put inside the Nintendo on a cartridge.
And if I zoom in on that sticker, I'll see, oh, this is FCC compliant, blah, blah, blah. This device may not cause harmful interference. And 2, this device must accept any interference that may be received, including interference that may cause undesired operation.
I take that to mean, also, desired operation. So this behavior is required by law. So that was giving me some hope.
Let's talk briefly about how it works. I know a lot about this topic. And if you want to talk over drinks, please. But I've simplified it here so that we can get through the important stuff.
Basically the Nintendo's a graphics subsystem. And that's the main thing that I'm going to reverse emulate, is an array of characters. You could just imagine an ASCII matrix on textmode on your computer.
And then the graphics that are those characters, like the pixels that make up the letter A. Those graphics don't necessarily need to be letters. In fact, they're usually stuff like question mark box, but also letters. And those are 8x8 pixels. And they're 2-bit color.
So this is what a screen of Mario looks like. Those are the 8x8 blocks. And here it is in sort of sepia tone 2-bit color.
I can also apply colors. I've got 13 colors to choose from. And I can change which ones those are, although that's a little involved. But I can select regions of the screen to use one of those four palettes for their 2-bit color. And 2-bit color yields four colors, right?
So this is what this looks like. Oh, I always mess this slide up. These are all of the colors that a Nintendo can display. But I can't even show them all on the screen at once, because I can only get 13 colors in the palette. So I'm strobing through them.
This is what that looks like in Mario. The color regions are those larger blocks. And this is the reason why graphics appear to not be those 8x8 things, but those 16x16 things, like question mark box, is you can only change the color resolution of the Nintendo that often.
But this also yields some great observations. I love this one. These things here, the cloud and the bush, are actually the same exact tile graphics with different colors applied to them. That's a mind-blowing, brilliant move to me. I really love that.
So I talked about color resolution. That's an apparent limitation in the Nintendo. But it's something that, by reverse simulating it, by flipping it on its head, I can actually get around that a little bit. So normally, 16x16 blocks often have the same colors.
But here's how the rendering actually works. You don't need to read this. It's actually rendering an 8x1 strips as the hardware is going through. And it will read the color bits-- the color attributes-- multiple times-- I guess 16 times per block that it renders. It reads that same color bits.
But if I'm actively choosing what to return for any given address, I can return inconsistent results every time it reads the ROM, or RAM, for a particular color. Which means I can change the color very often. I can change it for every one of these 8x8 strips, instead of at this large granularity. That's the kind of thing I mean when I say unlocking the abilities that the Nintendo does have, but sort of on paper, if you read the development manual, it doesn't.
Another limitation is the update speed. So here's how the Nintendo's normally rendering. Its NTSC scanning, 240 scan lines on your good old CRT TV, or this extremely fancy thing.
And then there's 20 VBLANK cycles at the end, where it's waiting for the CRT to refresh. And that end, which is only 1/13 of the time. This is more or less to time scale.
And that end is the only time that you can actually write to the PPU. So you can actually change it, for example, the palette. Or you can change that matrix that makes up the map.
So that's really a bummer, because in Mario, this is the amount of the screen that can change per frame. You can't repaint the entire screen by any stretch. If Mario moves too fast, the screen wouldn't be able to keep up, because you could only replace about two columns' worth of the screen.
Yeah, so I'm going to be able to get around that, because I can, of course, replace all of the bytes anytime they're read, including during the same scanline. So I should be able to do full-screen animations. You've even seen some of that already, now that I've spoiled, that this talk is using the very technology it describes.
Let's get a little bit down into some of the harder problems I needed to solve to do this, just because we're at a technical conference. But I'm going to try not to spend too much time on this, because I know I'm running-- well, there's a lot of material.
So here's an oscilloscope image of this Nintendo communicating with a cartridge in digital mode. And another weird thing I'm doing here is I've taken a reading of these traces. I've turned that into a computer image, then put it back into the Nintendo. And then those are flowing over those very same traces in order to render an image of the oscilloscope.
That's, I don't know. That's kind of a weird loop to me. Seems a little wrong.
And the timings here are actually pretty hard, even though it's supposedly kind of a slow thing. We're talking about nanoseconds. And so basically you have this read and write clock, or wait and write clock. And then you have addresses that come in, and data that comes out.
So first test is how fast can the Raspberry Pi do this same kind of thing? And the answer is that fast. And that is not really a lot of headroom. I'm used to orders of magnitude. Here it can do about three times faster than the Nintendo wants.
And worse, if I do the normal thing, which is when the read cycle begins, I read the address, and then write the data that corresponds to that, and I hand optimize that assembly code, and I turn off memory barriers like a cowboy, it's still late. It does not happen in time, and it completely doesn't work. It's really hard to debug, too, it turns out.
So here's an example of a kind of trick you do, which is a good life lesson. If you can't submit on time for the conference, submit on time for next year's conference. You'll be super early.
So what I do is as soon as the wait cycle begins, I know something is going to happen, and I write my data. And it takes 200 nanoseconds to get out onto the bus. And then I can read the addresses. And this gets me right there on time, with the major downside that my write happens before my read. So my write can't depend on the address that was read. So that's a little bit hard.
So what I need to do is I'm basically trying to keep up with the PPU, and predict where it's going on the screen, and then have the right byte cued up. And then I adjust my guess of where I am based on the read that comes in. And this is the main reason. This timing is the main reason why you're getting all that jank on the screen.
Of course, it's not really a digital device. This is a fantasy. This really looks more like that, which is itself another fantasy. It's an analog device. And I can't tell you how wrong my fantasy schematic is.
All you EE geniuses out there are really, I'm sure, laughing it up inside. But if you do this, you'll actually break the Nintendo, probably, and your Raspberry Pi. One of them is 3.3-volt CMOS. One of them is 5-volt CTL.
So you're like, oh, I know this. I'm going to use a resistor, uh-huh, and a transistor. And I'll put a 0 and a 1. But that doesn't work. It's totally, totally wrong. I was not knowledgeable about electronics, I guess. Because it's a shared bus. And, in fact, addresses and data use the same bus.
So you learn about tri-state buffers, and how this stuff actually works, which is cool, and I could explain to you now. I think I understand it. But that turns out that, oh my goodness, even though I built this thing out of transistors, it still takes 180 nanoseconds to discharge.
How is that possible? Isn't everything just made of transistors? Aren't they the fastest thing in the world?
Well, it turns out in the '60s they invented a way of preventing these things from saturating too much. And that's called the Schottky transistor. And there's this whole line of successive improvements to TTL. And if you look in here, it's original low-power, high-speed, Schottky, low-power, advanced. And then the skinless low-power Schottky is probably what we're going to invent next. That was a Engelbart joke.
But right about the right era for the Nintendo you'll see, oh, low-power Schottky, which has about the right timing characteristics that you're measuring. And then you're like, oh, wait. I'm looking at the Nintendo motherboard, and here's a chip on it-- SN74ls. These letters mean something?
And so you think, oh, I know what I'll do. I'll start copying off the Nintendo design. So I'll put some of those goombahs on my board as well. But, OK, so all you EE geniuses smugly out there, what is this component that you're copying off a-- you know it's a trick, now. You're copying this off a Nintendo circuit board.
It looks exactly like a resistor. It's got the stripes. It's exactly a resistor package. But it turns out it's a capacitor. In Japan in the '70s they would do this because they had machines that could place-- pick a place-- resistors, but not capacitors. So you can easily be led astray also by studying-- I know-- by studying those that came before us.
So I built this thing. This is the more the real-world schematic. Actually, it's the Gerber files. It's kind of complicated. I called it that PPU. So it's got a good logo. But actually I made a mistake in naming this, because I don't know how to pronounce words that start with two P's. It's like maybe P-P-Puppy.
And this is what it looks inside the actual cartridge. That's a picture of this actual cartridge that I've put inside the cartridge to display through the Nintendo onto the screen, the picture of the thing. You can see the dog logo on that.
It's got one of these guys on it. And this is a real slap in the face, because even though it's a stupid little copyright chip, you've got to harvest these from real games. So every time I want to make one of these, I've got to buy something on eBay, and take it apart, and desolder it, which is a pain. Often break it.
And Nintendo games are getting kind of expensive now. The cheapest Nintendo games to harvest for their components are more expensive than the cheapest Super Nintendo games, which I consider to be, I guess, I consider that to be wrong.
So there's also this big chip that's the ROM for the CPU. I mostly am doing this for the graphics subsystem, but there has to be a program that does something. And I'm going to tell you about that. And then we're going to get onto a demo-- another demo.
So the CPU's job is going to be to update the palette-- the global palette-- for the screen. And it's the only thing that can do that. The palette isn't read out of the same memory that the graphics are read out of. So it has to be done by the CPU.
And it has to do the input. So I'm controlling this, because I didn't want to rely on those clickers. Those are having a lot of technical issues, don't you think? This is really solid in comparison.
So here's basically how that works. Normally the screen is rendering scan lines, and it's the PPU talking to my board. Just that. CPU is just sitting there waiting for something to happen.
Then an interrupt happens. Interrupt tells CPU, I'm in VBLANK. You can now communicate with the PPU, which the PPU is where I'm going to store the palette.
But I also need to get what the next palette is from my board, which tells it that thing, I should have a red and blue and a green on this screen. So I need to communicate it with it. But the only thing I can do is read bytes out of it.
So I do this trick called knocking, which is I'm going to read in a specific sequence to tell the board, hey, I'm not rendering now. I'm trying to sync up with you. I'm the CPU.
So what I do is I just basically read. And remember, this is annoying, because the PPU-- sorry. My board is sort of predicting what's happening. It's not actually responding to these in real time. So it takes some steps.
But I read in descending memory locations, which should never happen, since the PPU renders left to right. I do that until I kind of sync up with it. And once I'm synced up, now it can stream data to the CPU.
And what it does is sends what the palette should be for the next screen. And it also sends the scroll position, because I'm going to do some tricks later with modulating the scroll position to improve the graphics. Improve the graphics.
So that's great. I can send the palette. By doing this trick of knowing that behind the scenes there is something that's smart, it has hysteresis about the order of reads.
So mission accomplished on that. But I also want to be able to report inputs. The CPU can read the controller. So I want to send data to the board.
Turns out this can be done the same way. So once I'm synced up, the addresses, you normally think of a read as a one-way operation. You give the data to me. But if I encode in the addresses that I read some information, then I can pass it to you, and you can send me data that you don't care about, if you like.
So basically just jam the joystick bits. There's 8 bits per controller. 8 bits. Makes sense. Inside those reads. So that's how that works.
Good. Well, bad actually. Because I used up basically an entire VBLANK doing this. I have 2/3 of it devoted to that process, which gives me 514 microseconds to do interesting stuff on the Raspberry Pi, which is not really enough. It's enough to do this slide presentation. But I wanted to do more interesting things to get my joke done.
So we are going to switch-- oh, I'll show you the slide that tells me to switch. We're going to switch to another cartridge. This cartridge, I've got it in an anti-static bag, because this one is really, I don't know if you can see. But it's like it didn't quite fit in there. So I thought this was not quite as faithful to the dream. But it is, I think, a pretty good joke.
So if it boots, which is definitely risky. Also vote for me for riskiest talk. Weirdest and riskiest.
Right. Unfortunately, I was using up the entire CPU. This is good. This is a good sign. I used up the entire CPU.
But I want to be able to do something else interesting. So here's a multicore Raspberry Pi. It's the highest end Raspberry Pi. That's why it doesn't fit. And also, I needed to upgrade the power supply. And it gets kind of hot, so we can't-- I don't think we can safely run this for that long.
[LAUGHTER]
And it might be F-sucking right now, FS checking. Well, hopefully it will work. Anyway, I'll tell you about what it is. And then maybe we'll get to see it.
So it's a multi-core system. And the reason I didn't just do the talk off this was that I was unable to get it to-- so in the talk version, the talk cartridge-- oh, good. Well it's got the wrong background color. But that should be OK.
In the talk version, I basically disable Linux. So when I'm running, I tell the low-level hardware to turn off the timer interrupts. And then Linux can't ever come back, which is really good, because then I can be hitting these micro-second or nanosecond deadlines.
On this one, unfortunately, because I have two cores, I couldn't figure out a way to turn that off without disabling one of the cores. And so Linux is basically waking us up every second, be like, hey, do you want to schedule anything? I was like, I was doing something. Every second, like that. So that's what's causing the blinking.
[LAUGHTER]
But I hope you'll think it's worth it. So I'll press start.
And then here is my joke. I don't know why the background is pink.
But what we have here is the launch title for the Super Nintendo. It's the 16-bit successor of the Nintendo, running-- oh, I'm glad you appreciate it-- running inside the Nintendo on a cartridge that I made. But this is an unmodified Nintendo. It should not be able to make 16-bit graphics and so on, at least at this level of fidelity. So I can play this.
One weird thing is that the controller buttons, I found this out. So I don't have as many buttons as the Super Nintendo has. But they actually line up. Up is up. Up on the Nintendo is up on the Super Nintendo by some, I don't know, coincidence. Or maybe there's just some laws about that. And the FCC of Japan. Man, I really wish the background wasn't pink.
So what else to say about this? Yes. So I'm doing a bunch of tricks. Actually the hardest part about this is not-- oh, there's a lot of latency. Ooh-ooh. This may not be a 100-- oh.
[LAUGHTER]
I was going to say, this is probably not going to be 100% speed run. I'm going to give it one more shot while I talk.
So one of the tricks I'm doing is, the Nintendo only has those 13 colors. So I am switching the palette and the scroll offset a lot in order that every other frame-- I need this for safety-- in order that every other frame-- oh, my goodness. Yeah, you can't tell, but there's maybe 500 milliseconds of latency between my presses. It's not normally like that. It's just this cheap '80s system that this opera house has.
[LAUGHTER]
I'm just kidding. I'm just kidding. I love you, AV folks. Everything is working really great.
Yeah, so I'm vibrating the screen back and forth at 60 Hertz so that I don't get the same sort of 8x1 grid that the Nintendo sort of forces upon me. And this gives me smoother both color resolution and sort of pixel resolution.
So OK, that's that. I hope you understand this, this, the improper hierarchy, how this leads in. I considered doing a quine, a quine Nintendo. And that would look like this. This is where I'm running a Nintendo emulator inside the Raspberry Pi, and then processing all its graphics.
Oh, you like that one better. Ah. I actually thought that the Super Nintendo one was a better joke when I was thinking about it. But now I know. The clapometer never lies.
I want to play, but I know it's boring for you to watch that. So, oh, my, the graphics have really gone awry. Well, let's just get back in the talk. I'm almost done. This is the very end.
So I could reverse emulate the CPU. And you could unlock some better. You could make the CPU faster, basically, by streaming it, only constant reads and writes. But I want to just connect this back, just deconstruct. So we got to have some kind of connecting it back to big ideas.
So this technology is good, perhaps, for nostalgic hardware, making nostalgic hardware, bring it into the modern era-- calculators, the Nintendo, Super Nintendo, Nintendo 64, whatever-- the Game Cube.
But maybe there's some other uses. What's another piece of hardware that we have nostalgia for? How about the human brain?
I feel like--
[LAUGHTER]
I feel like we should think about these things, because AI, it seems obvious to me that we will succeed at AI. And that's maybe not a good thing. I worry about that-- not only because it could destroy us, and all of society, but I also I really don't the idea of feeling left behind.
So we have AIs today that can play chess so well that it's not even fun to play at them, just as one kind of boring example. And I don't want to live in a world where there's these ultra smart, really cool robots, that are like, oh, you invented reverse emulation. Look what I invented. And then they show me this way better stuff.
So I would like it if we can somehow figure out how to make use of our legacy hardware in a future world. And one of the things I like about this thing, this talk, which is mostly a joke, is it taught me a little bit about what it might feel like to do that.
So I'm not talking about uploading your brain into the cloud or something. That would be emulation. I'm talking about reverse emulation.
So what if I replaced part of my legacy hardware, like my memories, with something that was vastly better than the actual hardware that I have? And obviously I haven't solved that by any stretch. And I'm not even working on it, because you can see how bad I am at electronics.
But if we were to solve that, if we understood the sort of logic family that is the human mind, and we could connect that up with, I would probably splurge for the high-end Raspberry Pi, because it's like my brain. Maybe I could have computer super powers.
And maybe that would feel like the following. I'm trying to understand what are the prime factors of this number, right? It's a big, boring number. Computers, this is easy for them. For us, it's a hard problem; sort of intractable problem.
So maybe I might be able to make it feel like I just know all of these things. Everything is just a part of my rote understanding of facts.
So I look at that number, and then I think, oh, I'll just sing myself the ABCs. (SINGING) A-B-C-D.
But then this board that I've embedded in my brain picks up. It notices this knocking. And it thinks, oh, I'll just now replace his memory with the answer to his question. (SINGING) A-B-C-D 472 866 02.
We'd also have to figure out how to make it rhyme, otherwise it wouldn't be very satisfying. But I think that, I don't know, it might work. And that tickled my fancy when I was working on this thing.
So that's a good thing, except we have a problem, which is that it is outlawed. The galaxy federal police in M510-- the year M510 made an order to destroy the Metroid of planet Symbath, and the mother brain, mechanical life brain. So we're also going to need to pass some new laws in order to do this. But otherwise I think we're in pretty good shape.
That's my talk. Visit me online at Tom 7, wherever that is-- Twitter or Google or something. And yeah, let's enjoy lunch. Thank you.
[APPLAUSE]