I'm curious is it a good strategy to prebuild binaries (install packages) and use chef for config management only. Theoretically it should speed up init process when we need new machines (traffic spike), especially if we use clouds like AWS where we can create our own AMI (images), at the same time it could be not flexible in terms of how other cookbooks work..
I'm looking forward to your thoughts on the question along with best practices, experiences and ideas!
Both ends of the spectrum have their pros and cons, as does the area in between.
A. Use a standard, public AMI with dynamic system configuration at boot
(+) Does not require learning how to build AMIs.
(+) Does not require learning how to manage a stable of historical AMIs.
(+) Easy to change and test the system configuration by editing the config and running a new instance.
(+) Easy to manage across regions, accounts by sharing config file(s).
(+) Easy to run variations of the config by tweaking the startup config rules.
(-) Initial startup of a system can be slow, or very slow, depending on how much work your startup configuration process needs to accomplish.
(-) Requires understanding how to automate startup config, especially if used with Auto Scaling or Spot instances.
B. Pre-build custom AMIS
(+) Initial startup is fast as no config steps need to be taken.
(+) Works well with Auto Scaling and Spot instances.
(-) Requires learning how to build and manage AMIs effectively.
(-) Requires building a new AMI every time the desired config changes.
(-) Requires building a new AMI every time OS security patches are released.
(-) Long edit/build/test loop when changing config
(-) Multiple AMIs required for different regions, different accounts (unless shared), different variations of configs, different instance architectures.
C. Something In Between
You can get some of the benefits (and some of the drawbacks) of both strategies by doing something in the middle: Create a base custom AMI that has most of what you need for your instances, then run a dynamic configuration at run time that updates and completes the configuration, applying any specific variations needed.
This speeds up the boot time, possibly by a lot, but also gives some flexibility and not having to rebuild AMIs as often.
Recommendations
Start by creating automated scripts (or Chef recipes) that will take a public AMI and configure it to your desired system.
Use the automated config with public AMIs (approach A above) for as long as that works for you.
When and where you run into startup performance issues, use the automated config to create custom AMIs (approach B above). Automate this process as much as possible so that you can run it as often as needed.
If you have multiple system variations on top of the same basic base, build a custom base AMI and use approach C above to run the different system flavors.