If you’re like me you’ve been keeping abreast of the recent developments regarding the fail0verflow team’s reverse engineering of Sony’s root signature key. This feat allows the generation of signed homebrew code which can run natively on the PS3 without the need for an existing jailbreak which bypasses the signature check. However, beyond the implications of this feat how did the fail0verflow team accomplish the impossible task of reverse engineering a private key from publicly available data? The answer lies in Sony’s botched implementation of Elliptic Curve Digital Signature Algorithm (ECDSA).
All code which executes on the PS3 requires a valid signature in order for the hardware to allow its execution. In the case of SELF (Signed Executable and Linkable Format) executable files Sony requires a signature within the file to be present which is an ECDSA signature of the file’s header utilizing Sony’s root signing key as one of the private variables. Sony’s crucial mistake comes in their implementation of the ECDSA algorithm which requires that all signatures be calculated with some unique random number k. Instead Sony used a fixed value for k across all of their application signatures which in turn has rendered the ECDSA algorithm effectively useless.
In the case of ECDSA when the random seed k is constant across more than one signature ECDSA hashing function can be solved for the private key d in the form d = (s*k – z) / r where s, z, and r are either publicly known values or are calculated as part of the ECDSA algorithm from publicly known values.
With the private key d now known SELFs may be generated which pass the security validation on the PS3 hardware and may run as native code without restriction. Furthermore, with this method duplicated across all levels of the PS3′s security layer less scrupulous members of the community may use the same method to trivially generate the private signing keys for game encryption, firmware validation, and even the system’s bootloader.
So with PS3 custom firmwares and native homebrew already starting to show up where does Sony go from here? Only time will tell. However, looking back you can say that you fully understand how it all began.