CON Exploit
The CON exploit was an oversight in Rock Band 2, Rock Band 3 and Rock Band Blitz for the Xbox 360 that allowed for loading of custom songs and pirated DLC on unmodified consoles. The oversight was caused by Rock Band 1's export feature using console-signed (CON) save game files to export songs to future titles in the series, meaning that once the creation of CON files manually via a PC was discovered, it was possible to have the game load songs never intended to be played in-game. This also worked online, allowing you to play custom songs with friends so long as they also had the same song and that the song ID was valid for online play.
Official Fix
The exploit was fixed in Rock Band Blitz and Rock Band 3's Title Update 5 by including a full list of SHA1 hashes of Rock Band 1 exported MIDIs in the executable. If the SHA1 of the MIDI file did not match, or if the CON package referenced a song path that wasn't in Rock Band 1, the game would display a "dirty disc" error upon loading the song and promptly exit. The fix was rolled out on newly produced discs of Rock Band 3 - discs with the Mad Catz logo are pressed with Title Update 5 included, and thus are impossible to downgrade to older versions.
Files packed in the Xbox Live-signed (LIVE) file format did not have any of these signature checks, thus users of modified consoles (such as an RGH or JTAG) could fakesign custom songs in that format and continue to use TU5. RB3Enhanced for Xbox 360 includes a patch that removes the check for RB1 MIDIs when loading through CON files, allowing regularly packed customs to work on modified consoles.
The bug was never fixed in Rock Band 2.
Bypassing
It's possible to have customs loaded on a retail system with TU5 by abusing the ability for the game to handle instrument upgrade MIDIs. By using a stock RB1 song's path and MIDI file, and placing the custom song's MIDI file as an upgrade, the game would allow the song to play. However, this method is not perfect and comes with some quirks, such as the tempo map being loaded from the original RB1 MIDI. This bypass also does not work on Blitz.
This bypass was originally discovered by PikminGuts92 on ScoreHero.