I want to install some package with not very useful dependencies (for example I want xorg-x11 without any xterm, xclock etc) and when I try to put dependencies in package.mask portage just complains about masked packages instead of ignoring them.
I think I had a solution for this but I'm forgetting it every time.
/etc/portage/package.mask will keep packages from being installed if they are optional. Otherwise they'll just block installation.
You probably want to find use flags for what you're looking for.
In the case of xorg-x11, it's a meta package that pulls in lots of stuff. You could install the dependencies by themselves if that's all you want.