I need to connect several servers on various public IPs together to form a virtual encrypted network, where ports and services on the servers are accessible only to those servers in the network and where all communication is authenticated and encrypted.
One obvious solution is VPN made of IPs in one of the private ranges, say 10.1.0.0/16. But this requires assigning each server additional IP address, which I'd like to avoid.
Is it possible to use e.g. OpenVPN or other software to create such network of public IPs, without assigning new IPs from private range?
No, if you use OpenVPN or similar, you will still have to assign other IP addresses for the VPN connections. The only way to do it that technically answers your question would be to assign additional addresses from part of the public IP V4 space that you don't ever expect to have to route to, and just treat that part of the public space as if it were private. But while it would be an answer to your question as written, it would also be silly. Just go ahead and assign additional IP addresses, it's not like you're in danger of exhausting the 10.0.0.0/8 address space.
Using IPSEC, it would be possible to set this up. It may also be the case that OpenVPN can do it, but I am not sure how one would go about that.
For IPSEC, this link has a very nice overview: https://libreswan.org/wiki/HOWTO:_Opportunistic_IPsec
I am not sure how portable that setup is to other IPSEC solutions (notably, for Strongswan, this suggests such a setup may not be possible).
That being said, Opportunistic Encryption is not the only solution to your situation: as long as there was a policy applying encryption to packets between your nodes, then you can drop the Opportunistic aspect of things, and you would be using regular IPSEC.
So yes, I suspect you could have traffic between any two nodes you operate encrypted, and leave everything else alone, as well as avoid adding more IPs to your hosts.