I wonder what is the general preference and where does it come from. Having used FreeBSD actively for a couple of years I'm leaning towards Gentoo, but I've had an unpleasant experience of wasting valuable time because Gentoo was installed on a really old machine with unfunny build times.
Had it been something prebuilt package based, less time would be wasted. (I know FreeBSD has binary packages, but problem with these are that they're seemingly unmaintained past a version release and the only way to get fresh stuff is to compile it yourself. Don't know about Gentoo really).
What's your take on this, have you had any experiences that made you switch from source based distro to something else?
EDIT: think I should clarify this recent "experience of mine"
The story goes like this. Having changed jobs, I've got to maintain a rather old machine which hosts LDAP (OpenLDAP) with office user login information. It came to me having to reboot the beast (it wasn't rebooted for 8 months). After the reboot, OpenLDAP didn't came online. It seems like slapd and some other binaries got removed somehow while the system was running. After building ldap package for the first time, I wondered why I got no slapd binary (took 15mins). Some time later I've tracked the issue down to "minimal" flag being on by default, which builds just the libs, no server binaries. End result - ~1hour of crippled office productivity and colleagues using excuses as "I didn't do that because our main server was down".
--
Really appreciate your answers, but so far no compelling arguments to use source based vs prebuilt package based. Storage space is not an issue these days.
I maintain more than a fair number of Gentoo machines.
Not because we care for childish funroll-loops speed tuning. What we care about is the flexibility of installing precisely what we want, not every package feature and dependency that a contributor thinks we might one day want. We are perfectly comfortable with how Linux, compiling and libraries work. We don't want everything to be abstracted away from us into a black box.
You should choose to use what you feel most comfortable with. Here are some details that may help to allay your fears of Gentoo specifically.
Build times are relatively negligible on modern hardware. Even more so if your machines are of N+1 redundancy.
You can be picky about how portage behaves, such as placing
MAKEOPTS="-l 1.0"in your make.conf to ensure that new builds backoff when the load average is creeping uncomfortably high.You can use binary packages from Portage if you'd prefer. The mirrors provide a number of common packages or you can roll your own with
--buildpkgonly.If you have a large number of machines then you can benefit from having a nominated build host or distributed compiling.
The latest stable versions of Portage that are now in use are much more unlikely to leave you high and dry when performing upgrades. Tales of conflicts and blocks are pretty much a thing of the past.
If you are looking after so many machines that upgrades become painful then you should be looking at Puppet/BCFG/cfengine anyway ;)
Update in response to question edit:
Whilst I empathise with the situation you describe, it is not something that is symptomatic of source-based distributions or indeed prevented by using a package-based distribution. It is the result of:
Truth be told, I haven't extensively used any other Linux distros for quite some time now. For this reason, if you were to place a Debian machine (for example) onto my lap and tell me that OpenLDAP wasn't working after a reboot, then I too might spend 15 minutes or an hour to resolve the issue. Not because I don't have a good understanding of Linux or that Debian isn't a good OS, but primarily because I don't recall the intimate details of Debian's RC scripts or package system. This is why internal documentation is essential within an organisation. It should serve to jump-start the unfamiliar and to fill in the gaps of everyone else. Even if they do know everything.
Just a quick note specifically about Gentoo. Portage's USE flags are incredibly useful and "minimal" is something that I use frequently. I don't for instance want the server binaries of a package to be installed on a machine that will only ever be a client during it's lifetime. Having them unnecessarily present may increase complexity or even be a security concern. It's never an issue of space. You can see what USE flags and dependencies a package is going to adopt (and which it isn't) before it starts compiling by using the
-avarguments. So you shouldn't get an surprises.GNUix> Sorry, snap!
I won't comment on if you should use Gentoo or not - or if building from source 'is worth it'
What I will say is that I use Gentoo and Ubuntu. I used to use Gentoo for all my GNU/Linux machines but decided that Ubuntu on my desktops was easier to manage. I am on the side of admins that think using Gentoo for a server is a great idea -- and use it for almost all of my personal servers.
Apart from all that, what I like more than building stuff from source or tweaking CFLAGS is Portage. Portage is simply hands down the best package and system management tool I have ever used for GNU/Linux. I can create an exact environment with just the things I need and not one thing more and I can do it with out going outside of my package management system. I don't have to have postfix installed simply because some packaging dude decided that it's 'necessary'. I don't need mono installed because I want to run Gnome, I simply don't install Tomboy. Quickly browsing the installed packages on this Ubuntu desktop shows that it has bind installed? Why? It serves no purpose, I'm not running a DNS server on this computer and I certainly don't need the documentation for it, but it's here.
The downside... time. It takes not only the time to build packages but to keep the system maintained. I can be realativly sure that I can run a sudo aptitude update {upgrade} and not have a care in the world... On the other hand you have to be very careful of the small details when you update a Gentoo machine and make sure that you need (and are prepared) for what it wants to do. The 'straw that broke the camels back' per say of why I switched to Ubuntu for my desktops was an upgrade of udev that borked a setup I had -- rather than try and fix it I wanted to get the work I was trying to do done. So I figured I really don't care if bind documentation is taking up space on my desktop because I have the space.
On my servers, however, totally different matter. I want to control everything about the environment, the packages that are installed, every little detail I want to be in control of -- and Gentoo let's me do that with ease.
Life is too short for source based distributions. I have more important things to do. Like using the computer.
You can always use a combination of both methods - use a pre-compiled distribution as a base and have the advantages of a good working integrated environment and fast-painless updates while using self-compiled software where one needs to be at the leading-edge of the development of that software. Either by having some self-discipline (using configure with the prefix option) or by packaging the source yourself. You can always "reuse" the build instructions (rpm spec file or debian subdirectory for example).
I wouldn't use the build mechanism on production servers directly. From my Gentoo days I know that there's a possibility to have a build host and create the packages needed on that host so that you can burn cpu cycles on a dedicated host with the compile flags suitable for your needs.
Overall there isn't too much difference, nearly all of our servers run Debian but we ended up having a build host anyway. We needed a few software packages that weren't available as debs so we set up a reprepro and a buildhost for our needs, the buildhost is pretty much unmaintained (in terms of "does it work") because redeployment is just a matter of a few minutes since we can netboot it and it will automatically redeploy itself.
Either BSD ports/Gentoo/Debian it doesn't matter that much the real profit comes with using a system (as in collection of services on different hosts that provide business value) that will minimize wasted resources and is maintainable. We chose Debian because security updates are a no brainer and they are pre-compiled so that we don't have to "waste" time rebuilding packages whenever a security update comes out. That is about the only drawback I can think of for source based vs. binary based distros.
I was a avid Gentoo user for my MythTV machine, but i've recently switched to Debian due to the time required to upgrade a Gentoo machine. Fine you only have to really upgrade every 6 months or so but as you mentioned the compile times can be a little too much.
The best of both worlds would be ArchLinux, which has binary distributions but a simple build system to allow almost Gentoo style building of packages. You can pick and choose what you want to compile by hand.
For a production system, personally, i'd stay clear of source distros and stick with something stable and possibly supported (depending on your technical staff), either Debian or RHEL.
Quite frankly I can't think of a good reason to use a source based distribution other than for educational purposes (maybe).
My distribution provides source packages for all their binary packages so I have the freedom to build what I want when I want and more importantly I have the ability to not have to build everything all the time.
Gentoo is amazing not just beacause of the advantages of source based distributions, but because every step of the way is designed for customization.
config files are never updated, copies with the prefix ._cfg000 are place in /etc and a message is displayed for you to go move the new config files in place of the existing ones.
I find that, with a little effort, gentoo can make even complex things very easy to do.
I'm going to be a Johnny-come-lately because I searched for this topic, and I'm sure someone else will too.
We use Debian on all the live servers we can because:
Ease of systems administration is #1 in our books. An easy system in the hands of an expert equals high reliability and availability.
I haven't used Gentoo for a few years, but my limited experience with it brought me to the conclusion that the (barely noticeable) speed increases gained by compiling the packages were far outweighed by the disadvantage of the time taken to build the system. I've always used precompilied package-based systems since then. I know that, personally, I wouldn't like to spend so long compiling sources on servers - especially when needing to upgrade packages over time.