Is it safe to allot an IP address to a computer, and not running a webserver on it?

Sure. TCP/IP can be used for anything, including connecting to databases. For instance, you might be interested in the MySQL wire protocol.

Of course, your database server must be properly secured. You should be especially careful if it is accessible from the outside Internet.

See this article for how to secure MySQL. Similar (but distinct) advice applies to other database systems.

Best practice as I've known it is to stick a firewall between any server & the Internet, NAT the public IPs to private, and open the necessary ports ONLY. Absolutely lock that server down as described, but at the least, this will save your DB server from having to process all the random port scanning & scripted attacks.

If I understand mattdm's clarification to your question, then you are currently running DB servers in Amazon EC2 on instances that do not have public IP addresses. Not having any experience with any of Amazon's "cloud" offerings, am I correct to assume, then, that you access your DB servers from your other servers via some kind of Amazon-provisioned private network?

Anyway, keeping my (possibly way off-base) assumptions in mind, if your current setup works for you without assigning public IPs to your DB servers, then why change it? Giving your DB servers public IP addresses will make them (potentially) accessible from the internet at large, which will introduce security concerns that you will need to address. Since it is very easy to get security wrong, my answer to your question is No, unless you've already got a well-thought-out security infrastructure built up around your servers.