How To Deal With Frequent And Constant PHP/MySQL Version Updates

Ok - so let's boil this down:

The primary issue you're facing is that you need to run a newer version of PHP/MySQL, but by the time your QA team has tested the software on the new system, that version is out of date.

That's not a failing of your team; that's a failing of the QA team.

Your QA team should have automated test cases. Your developers should be providing test modules as they go along. E.g.:

• Developer builds pro-rated invoicing system
• Developer provides a test interface for pro-rating (i.e. accepts all the details needed to test the system, but does not actually generate an invoice. Or maybe it does, but in a staging system).
• QA team devise a series of tests to run against the routine. What is the input, and what is the correct outcome? E.g. $50/month pro-rated for half a month == $25.

This last step should be fully automated. It should take a matter of minutes or hours (depending on the complexity of the software or how many tests there are) to determine any errors. E.G.

• PHP changes its floating point maths from normal rounding to cieling (always round up)
• Test case of pro-rating at $5/month for half a month
• Correct answer: $2.50, but test case returns $3. Automatic fail.

Now, I know that none of this really answers your question, so if your QA team can't be fixed, then what are our other options:

There's no one correct answer to this. Personally, what we do with our software is your last option - we accept all updated and service packs to our runtime environments, but we do not change versions without a thorough and complete QA analysis. Although, the software we use in our runtimes (which is MSSQL and another back-end, not PHP) only release new versions every 18 months to 2 years, with updates and service packs inbetween, so there's plenty of time between major rollouts.

After QA tests on our staging systems, we deploy the update to our dogfood systems first. If we don't find any major issues, we roll the runtime updates out to our hosted systems. If we still don't have any major issues, we release notices to our self-hosted clients that they can install the updates if they wish.

Personally I cringe every time I get a pre-packaged virtual machine. Even if it's totally up to date (they never are, as you've pointed out) it's something that I then need to work on integrating into our network. And sometimes they ship with horribly out-dated operating systems or runtimes.

You have stated your question as follows:

1. Periodically, critical security flaws are found in components of your server stack, including PHP and MySQL.

2. When such flaws are discovered, the developers of PHP and MySQL release new versions. In the interest of security, old/insecure versions may no longer be available in the repositories that you use.

3. Your web application must be PCI compliant. When you use old, insecure versions of PHP and MySQL, your web application fails PCI security scans that must pass in order to achieve compliance.

4. You are never able to use the current version with all security fixes, because your QA team takes "a lot of time" (apparently months) to give the green light to use each security release of PHP and MySQL. In fact, they wait so long to approve security updates that you are never fully patched.

You then proceed to explore various possibilities for making it easier to install old versions of PHP and MySQL with known vulnerabilities. However, items #1-3 above are not negotiable.

Item #4, on the other hand, is negotiable. Ergo, your company's QA team is the problem. Security updates incrementing the version number by 0.0.1 (e.g. from PHP v5.4.2 to v5.4.3) are very unlikely to break your application.

You should install the new version in your test environment, run your automated tests, do minimal manual testing, and then roll it. As with any change, you need to have a back-out plan in place in case of unexpected problems, but... come on, you're talking about patches for known exploits! The security risk that you take by not patching (which, by the way, includes guaranteed failure of your PCI compliance scans) far exceeds any risk you take by patching.

Feature upgrades (e.g. from PHP v5.3.x to v5.4.x) may require more thorough testing, and for good reason: for example, 5.4.0 removed various legacy features and some old applications might need to be updated accordingly. Fortunately, you actually do get months to test these releases. This is exactly why security updates are provided concurrently for both the current version and the previous version.

It sounds an awful lot like your QA team is confusing minor security updates with actual feature upgrades.

Executive Summary:

1. Security updates that do not add/remove features must be applied immediately with minimal testing. It is unacceptable for your company's processes to subject minor security updates to bureaucratic delays, unless actual problems are found in the initial testing.
2. When it comes to feature releases that have a much higher probability of breaking legacy applications, your QA team can continue to take its sweet time.