CON Exploit
The CON exploit was an oversight in Rock Band 2, Rock Band 3 and LEGO Rock Band for the Xbox 360 that allowed for loading of custom songs on unmodified consoles. The oversight was caused by Rock Band 1's export feature using console-signed (CON) save game files to export songs to future titles in the series, meaning that once the creation and signing of CON files manually via a PC was possible, it was possible to have the game load songs never intended to be played in-game. This also worked online, allowing you to play custom songs with friends so long as they also had the same song and that the song ID was valid for online play.
Official Fix
The exploit was fixed in Rock Band Blitz, Rock Band 3's Title Update 5 and the Mad Catz Rock Band 3 re-release by including a full list of SHA1 hashes of Rock Band 1 exported MIDIs in the executable. If the SHA1 of the MIDI file did not match, or if the CON package referenced a song path that wasn't in Rock Band 1, the game would display a "dirty disc" error upon loading the song and promptly exit. The fix was rolled out on newly produced discs of Rock Band 3, and the digital download - versions published by Mad Catz are made with Title Update 5 included, and thus are impossible to downgrade to older versions.
Files packed in the Xbox Live-signed (LIVE) file format did not have any of these hash checks, thus users of modified consoles (such as an RGH or JTAG) could fakesign custom songs in that format and continue to use TU5.
Both Rock Band 3 Deluxe and RB3Enhanced for Xbox 360 include a patch that removes the check for RB1 MIDIs when loading through CON files, allowing regularly packed customs to work on modified consoles.
The bug was never fixed in Rock Band 2 or LEGO Rock Band.
Bypassing
It's possible to have customs loaded on a retail system with TU5 by abusing the ability for the game to handle instrument upgrade MIDIs. By using a stock RB1 song's path and MIDI file, and placing the custom song's MIDI file as an upgrade, the game would allow the song to play. However, this method is not perfect and comes with some quirks, such as the tempo map being loaded from the original RB1 MIDI. This bypass also does not work on Blitz due to that game not supporting Pro upgrades.
This bypass was originally discovered by PikminGuts92 on ScoreHero.