Here is the complete guide on how to do it :
http://forum.whmcs.com/showthread.php?t=17963
but for step 4 (SMTP in WHMCS) download this file :
http://www.adminsehow.com/uploads/class.zip ( it is the customized PHPMailer v5.0 )
and rewrite the class.smtp.php and class.phpmailer.php in the includes folder of WHMCS.