No More Heroes.
I'm relatively good at the game now, but it's probably the longest learning curve I've been through in, like...ever. The worst one next to that is Viewtiful Joe, and it's for the same reason: learning the combat system.
Other than that, probably Rayman: Raving Rabbids. My cousin had that game, and I played it...and good God, I did not realize all the mini games were so damn hard.