Can I use a custom email server to send mail from the portal?

Yes, this is possible.

You would have set the fields for the host, port, possible encryption, username, and password under 'Settings' → 'Platform' → 'Settings' → ‘Emails’ → ‘Email server’