Basically, the ps3 has a hard outer shell, but once you break through the outer layer there is absolutely nothing stopping you…
The hypervisor does absolutely nothing except protect itself and abstract out some of the hardware. Once you gain control of the level 2 kernel it doesn’t throw up any road blocks. It will allocate the memory in anyway you ask it, as long as none of those pages belong to the hypervisor. It doesn’t enforce wx. It doesn’t check any code you ask for is has been signed. It doesn’t continually hash the code running below it to make sure it hasn’t been compromised.
IBM have essentially re-tasked one of their mainframe (Yes, IBM still make mainframes) hypervisors that was originally designed to abstract the hardware and allow mainframes to run multiple operating systems at once and sold it to Sony.
And then there are the isolated loaders which Sony are so proud of They are so isolated that the have no way to check the integrity of the system that they are running on. All you have to do is ask the hypervisor to load it for you and then you pass the stuff you want decrypted in 4KB at a time. From there you can copy it back out, patch it however you want and start executing it.
You can even pass the hypervisor your own custom isolated loader, like one from a newer firmware version where they have changed the keys. The hypervisor will load it and you can now decrypt games that were not meant to run on your firmware version.
The design of the security system is a complete failure.
There are a few reasons that it wasn’t cracked a lot sooner:
- Marketing. Sony talked up their security system so much that it scared a lot of people from even trying. And with the hard outer shell, people expected the inside to be just as well protected. When the psjb was released and hackers started working out how it worked we were shocked to find that WX wasn’t enforced in the kernel (It is enforced in userspace) and that the hypervisor would let you pirate games with no resistance.
- OtherOS kept a lot of hackers from even trying, It already did everything they wanted. Most hacking was focused on getting 3d accleration working from linux.
- OtherOS provided a decoy for hackers. Most early attempts were focused on trying to elevate OtherOS to GameOS status or comprise the hypervisor and take control of GameOS that way. The hypervisor was designed to protect against exactly that kind of attack (and only that type of attack).
Once OtherOS was removed, people started focusing on attacking GameOS for the first time, and it didn’t take long for GameOS to fall.