Apache not serving files from where expected

Maybe because you have ServerAlias rule for domain.com with

*.domain.com

which will match ebooks.domain.com and serve from

/home/domain.com/public_html

My best advice would be use mod_rewrite with RewriteCond %{HTTP_HOST} but unfortunately I don't know enough to give you any more specifics. Take a look at mod_rewrite manuals/guides.

I'd have thought that the order was the key here, that apache would serve the first matching servername, but I did a quick test, and that doesn't seem to be the case. It always seems to match the wildcard, no matter what order they're in.

The best I could think of with one ip address is pretty much what Evgeny suggested, have one virtualhost and use a proxy redirect for books.domain.

<VirtualHost *>
    ServerName domain.com
    ServerAlias domain.info domain.net domain.org *.domain.com *.domain.info *.domain.net *.domain.org
    DocumentRoot /home/domain.com/public_html/
    <Directory /home/domain.com/public_html>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>
    ErrorLog /home/domain.com/logs/error.log
    CustomLog /home/domain.com/logs/access.log combine
    ServerSignature Off

    <Directory /home/media/books>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    Alias /home/media/books /books
    RewriteRule   ^books\.domain\.com(.*)$      http://domain.com/books$1 [NC,P,L]
    RewriteRule   ^ebooks\.domain\.com(.*)$     http://domain.com/books$1 [NC,P,L]

</VirtualHost>

The only way I can think of to preserve the wildcards and seperate vhosts would be if you had two ip addresses, and assigned the two virtualhosts to seperate addresses. I still feel there must be a better way to do this though, will be interested the other solutions.

I think Evgeny nailed it.

Good to know: The value of a NameVirtualHost directive has to match the content of exactly and neither should be a hostname. For example, NameVirtualHost *:80 must be used with . "NameVirtualHost *:80" must appear only once in a configuration.

As Evgeny noted books.domain.com is served from public_html as that is exactly what the VirtualHosts say ... your first VirtualHost ServerAlias is ebooks.domain.com to serve from /home/media/books. Make sure the *.domain.com is always in the last VirtualHost [which you have] as long as some preceding VirtualHost has a ServerAlias matching the domain you enter that VirtualHost will serve the response.

If you're looking for a more completely wildcard solution and you can re-arrange the directory structure like /home/media/books, home/media/ebooks etc then try something like this and get away with everything in a single VirtualHost:

DocumentRoot /home/media #make sure you have a default index page 
RewriteEngine on 
RewriteCond %{SERVER_NAME} ^(.*).domain.com [NC] # make sure this is a subdomain call 
RewriteCond /%1/ -d # is there a directory for this subdomain 
RewriteRule ^/(.*)$ /%1/ # request filename now /home/media/%1 - client URL unchanged.

Do you have a "NameVirtualHost *:80" or similar as a global config line?

That's usually what's wrong when virtual hosts just don't work in Apache.