To me, the physics of the situation makes this all the more impressive.
Voyager has a 23 watt radio. That’s about 10x as much power as a cell phone’s radio, but it’s still small. Voyager is so far away it takes 22.5 hours for the signal to get to earth traveling at light speed. This is a radio beam, not a laser, but it’s extraordinarily tight beam for a radio, with the focus only 0.5 degrees wide, but that means it’s still 1000x wider than the earth when it arrives. It’s being received by some of the biggest antennas ever made, but they’re still only 70m wide, so each one only receives a tiny fraction of the power the power transmitted. So, they’re decoding a signal that’s 10^-18 watts.
So, not only are you debugging a system created half a century ago without being able to see or touch it, you’re doing it with a 2-day delay to see what your changes do, and using the most absurdly powerful radios just to send signals.
The computer side of things is also even more impressive than this makes it sound. A memory chip failed. On Earth, you’d probably try to figure that out by physically looking at the hardware, and then probing it with a multimeter or an oscilloscope or something. They couldn’t do that. They had to debug it by watching the program as it ran and as it tried to use this faulty memory chip and failed in interesting ways. They could interact with it, but only on a 2 day delay. They also had to know that any wrong move and the little control they had over it could fail and it would be fully dead.
So, a malfunctioning computer that you can only interact with at 40 bits per second, that takes 2 full days between every send and receive, that has flaky hardware and was designed more than 50 years ago.
And you explained all of that WITHOUT THE OBNOXIOUS GODDAMNS and FUCKIN SCIENCE AMIRITEs
Oh screw that, that’s an emotional post from somebody sharing their reaction, and I’m fucking STOKED to hear about it, can’t believe I missed the news!
Finally I can put some take into this. I’ve worked in memory testing for years and I’ll tell you that it’s actually pretty expected for a memory cell to fail after some time. So much so that what we typically do is build in redundancy into the memory cells. We add more memory cells than we might activate at any given time. When shit goes awry, we can reprogram the memory controller to remap the used memory cells so that the bad cells are mapped out and unused ones are mapped in. We don’t probe memory cells typically unless we’re doing some type of in depth failure analysis. usually we just run a series of algorithms that test each cell and identify which ones aren’t responding correctly, then map those out.
None of this is to diminish the engineering challenges that they faced, just to help give an appreciation for the technical mechanisms we’ve improved over the last few decades
Is there a Voyager 1, uh…emulator or something? Like something NASA would use to test the new programming on before hitting send?
Today you would have a physical duplicate of something in orbit to test code changes on before you push code to something in orbit.
I still cannot believe NASA managed to re-establish a connection with Voyager 1.
That scene from The Martian where JPL had a hardware copy of Pathfinder on Earth? That’s not apocryphal. NASA keeps a lot of engineering models around for a variety of purposes including this sort of hardware troubleshooting.
It’s a practice they started after Voyager. They shot that patch off into space based off of old documentation, blueprints, and internal memos.
Imagine scrolling back in the Slack chat 50 years to find that one thing someone said about how the chip bypass worked.
Imagine any internet company lasting 50 years.
This is why slack is bullshit. And discord. We should all go back to email. It can be stored and archived and organized and get off my lawn.
It’s not Slack’s fault. It is a good platform for one-off messages. Need a useless bureaucratic form signed? Slack. Need your boss to okay the afternoon off? Slack. Need to ask your lead programmer which data structure you should use and why they’re set up that way? Sounds like the answer should be put in a wiki page, not slack.
All workflows are small components of a larger workplace. Emails also suck for a lot of things. They probably wouldn’t have worked in this case, memos are the logical upgrade from emails where you want to make sure everyone receives it and the topic is not up for further discussion.
To add to the metal, the blueprints include the blueprints for the processor.
https://hackaday.com/2024/05/06/the-computers-of-voyager/
They don’t use a microprocessor like anything today would, but a pile of chips that provide things like logic gates and counters. A grown up version of https://gigatron.io/
That means “written in assembly” means “written in a bespoke assembly dialect that we maybe didn’t document very well, or the hardware it ran on, which was also bespoke”.
I realize the Voyager project may not be super well funded today (how is it funded, just general NASA funds now?), just wondering what they have hardware-wise (or ever had). Certainly the Voyager system had to have precursors (versions)?
Or do they have a simulator of it today - we’re talking about early 70’s hardware, should be fairly straightforward to replicate in software? Perhaps some independent geeks have done this for fun? (I’ve read of some old hardware such as 8088 being replicated in software because some geeks just like doing things like that).
I have no idea how NASA functions with old projects like this, and I’m surely not saying I have better ideas - they’ve probably thought of a million more ways to validate what they’re doing.
There is an fascinating documentary about the team that sends the commands to Voyager 1 and 2 called It’s Quieter in the Twilight
Still faster than the average Windows update.
More stable, too.
Absolutely. The computers on Voyager hold the record for being the longest continuously running computer of all time.
Certainly better tested.
Well, they only had to test it for a single hardware deployment. Windows has to be tested for millions if not billions of deployments. Say what you want, but Microsoft testers are god like.
Windows? Hardware testing? Testing in general? LMAOOO
Yeah that’s what the updates are for haha
Interviewer: Tell me an interesting debugging story
Interviewee: …
Heh. Years ago during an interview I was explaining how important it is to verify a system before putting it into orbit. If one found problems in orbit, you usually can’t fix it. My interviewer said, “Why not just send up the space shuttle to fix it?”
Well…
Why do Tumblr users approach every topic like a manic street preacher?
There’s a significant overlap between theatre kids and Tumblr users.
That ven diagram is maybe 3 degrees away from a circle.
Man I can’t even get my stupid Azure deployment to work and that’s only in Germany.
I think the term “metal” is overused, but this is probably the most metal thing a programmer could possibly do besides join a metal band.
When I hear what they did, I was blown away. A 50 year old computer (that was probably designed a decade before launching) and the geniuses that built that put in the facility to completely reprogram it a light-day away.
Keep in mind too these guys are writing and reading in like assembly or some precursor to it.
I can only imagine the number of checks and rechecks they probably go through before they press the “send” button. Especially now.
This is nothing like my loosey goosey programming where I just hit compile or download and just wait to see if my change works the way I expect…
they almost certainly have a hardware spare, or at the very least, an accurately simulated version of it, because again, this is 50 year old hardware. So it’s pretty easy to just simulate it.
But yeah they are almost certainly pulling some really fucked QA on this shit.
I wont even upgrade the BIOS on my motherboard because im afraid of bricking it.
Meanwhile here on Earth, we need to login using two accounts to access Helldivers 2. And even got pulled from many countries. What a time to be alive.
Sony has agreed to remove psn requirements for pc users now
I just have to imagine how interesting of a challenege that is. Kinda like when old games only had 300kb to store all their data on so you had to program cool tricks to get it all to work.
No yeah, it’s like that plus the thing is a light day away, and on top of that malfunctioning on a hardware level. Incredible
It’s like you already have a 300kb game on a cartridge, but it doesn’t work for some unknown reason. Also you don’t actually have the cartridge, some randy in Greenland does. And they only answer emails once every 2 days or so.
People always underestimate the high level NASA works at. Everyone bitches and moans, especially Musk simps, about how long SLS took to make and its expense, but it worked right the first time. In the case of the Voyager spacecraft, they are working with tech so old, all the original engineers are retired or dead. NASA rocks.
I understand your point and completely agree that NASA has produced some amazing technological feats, but we could probably use a different example than the SLS to highlight their accomplishments. Even with supposedly repurposed rocket engines and technology from the Shuttle era, that project is billions of dollars over budget and years behind schedule. If you want to highlight how amazing it is that SLS has actually flown with all the political manipulations associated with it, then I’d probably agree with you in that sense. This is no criticism of the engineers, but to completely ignore the issues of this project as a whole, not just financially related, seems to be a bit disingenuous.
Here’s a good article from Berger talking about what the Government Accountability Office thinks of the project: https://arstechnica.com/space/2023/09/nasa-finally-admits-what-everyone-already-knows-sls-is-unaffordable
The budget wasn’t really relevant to my point. And it did work correctly the first time.
All I’m saying is you could choose a better example, which NASA is full of them.
But lets say I built you a car that already came with an engine and some other important things, just to make it quicker and cheaper to get that car in your hands. Unfortunately, you want me to complete work on the car in five different states and use components from those areas. Guess what, the car is now about $5 million over budget and 5 years behind schedule. Not only that, but we encountered issues during the first test that are going to require more fixes ($$$) and more delays for the second test.
In this situation, you’re saying it’s great, it ran correctly the first time because it went down the road and back, and budgets and timelines don’t matter. I’m saying ehhhh, not really - we’re over budget by millions, delayed by years, and there were issues, even though we repurposed stuff that was in a car that actually ran a few years back. It’s great we built the car, but the project itself isn’t something that I would showcase as my best work.
Great documentary on the Voyager team: It’s quieter in the twilight
It reminds me that there are still very intelligent and talented people within our ranks. A nice breath of fresh air.
