In order to implement SSO, I have been working with some IdP and a Shibboleth SP install without being able to answer this question.
On the IdP side I have a few metadata files that describes some applications. Those files can contain a certificate, but if nothing is given, communication works still fine. If I put the same files on the SP side, it still works fine even when putting some random characters as certificate.
Could someone explain to me really how it works and what is the purpose of those X509 certificates in those metadata file (IdP) side?
These certificates are the trust anchors. They allow you to verify signatures and therefore establish trust in the messages that have been exchanged. (This is very similar to trusted CA certificates that are configured in your browser for authenticating certificates of HTTPS sites.)
The SAML requests and responses should be signed (the responses at the very least). You'll find elements like
<ds:DigestMethod />,<ds:DigestValue />and<ds:SignatureValue />(although this may be simplified if you're using the HTTP redirect binding).When the SP gets a SAML response from the IdP via the browser, it must verify that the signature it gets comes from an IdP it knows and what signed using the IdP's private key; this signature can be verified against the IdP's public key in the certificate configured in the metadata.
For an SP, failing to do this makes the whole process worthless, and being able to accept any assertion without verifying the authenticity of the IdP's message suggests a configuration error.
On the IdP side, it's less important to a degree. It's only necessary if the IdP wishes to verify that the request really comes from an SP it trusts. It's particularly useful if the SP requests certain confidential attributes to be disclosed (that not all SPs should see), and possibly to encrypt those attributes in the response so that only this SP can read it.
This being said, Shibboleth can release these attributes through a back-channel (the attribute service), where a connection between the SP and the IdP is made directly (without message exchanges with the browser). Authentication between the SP and IdP should still take place in this event, but I think it might work at the transport level (e.g. client-cert authentication) instead of message level (via SAML signatures), I'm not sure. Either way, you would also need to set up trust anchors for this.
These certificates are there to ensure that the data exchanged between the idp and sp are:
These are security features of the protocol which one should not omit when deploying a production system.
In systems which are part of a federation x509 certificates are actually used (they are mandatory).
The X509 Certificates contain keys for public key cryptography, and an IdP key can be used for either signing messages from the IdP or encrypting messages to the IdP or both (although one wouldn't usually encrypt messages to the IdP).
Have a look at this stack overflow question for the difference between signing & encryption: https://stackoverflow.com/questions/454048/what-is-the-difference-between-encrypting-and-signing-in-asymmetric-encryption
Have a look at the Shibboleth documentation for examples of using keys in metadata: https://wiki.shibboleth.net/confluence/display/CONCEPT/MetadataKeyDescriptor
Whether or not a key is actually used for either signing or encryption depends on the profile of the messages being passed between the IdP and SP. That is to say, whether the messages are in SAML1 or SAML2 format, and which of the endpoints are used.