I have some basic question about SMTP (try not to laugh too much).
When i try to use SMTP via telnet to send email on google it requires authentication. Mail servers use SMTP to relay messages between them right? So how do they relay using SMTP without authenticating to each other? I queried for the MX of googles mail servers and tried initiating SMTP session via telnet, but it didn't seem to work, I just had a black screen, no HELO, no EHLO, nothing.
A common configuration is that email destined to addresses "inside" the organization does not require authentication, while "outbound" email does require authentication. This allows an organization to receive inbound email while not becoming a source of spam sent by unscrupulous random people on the internet.
This is often implemented by having completely separate sets of servers responsible for inbound and outbound email. Outbound servers are configured to require authentication in all cases, while inbound servers do not require authentication but will only accept messages destined for delivery inside the organization.
This seems to work as expected.
You have a slight misunderstanding of how email works.
When you send email from your email account/client (using SMTP) through your email provider to a recipient whose email is handled by another email provider your email provider requires you to authenticate because you are asking your email provider to relay your email THROUGH it to some recipient whose email is handled by another email provider. You are sending an email to a domain that your email provider is not authoritative for, hence the need to authenticate because you are asking your email provider to relay the email to another server.
Your email provider in turn makes an SMTP connection to the recipients email server to send the email TO the recipients SMTP server, not THROUGH the recipients email server. The recipients email server is authoritative for email for the recipients domain, therefore your server sends the email TO the server, not THROUGH the server. This is how SMTP servers send emails to each other. This is NOT relaying.
I might have misunderstood your question but it seemed to be geared more towards the general nature of how email relay/delivery works without passwords. I'll respond to that.
When a mail server is considering whether or not to relay an email message, it will typically be looking for a few key characteristics:
The source IP address of the upstream connecting server/client - In other words, where is the message being sent from (e.g. 60.60.60.60)
The destination email address of the recipient - In other words, who is the message going to.
(There are a lot of other things it can take into consideration but I'll leave those out for the purpose of simplicity).
If the email recipient domain doesn't belong to the mail server itself, the server will (quite rightly) be suspicious of the message and require some sort of "proof" to verify that the sender is a trusted source and not a spammer or malicious user trying to abuse the system. This is where the source IP (among other things) can help.
A simple setup would be for IP addresses of trusted upstream servers to be added ahead of time. So for example I could tell my email server that it's OK to relay messages from [email protected] to anywhere IF (and only if) the message was coming from a trusted IP.