Software evaluation 2.0 ?

I spend a lot of time evaluating software; for product reviews, to see which versions are vulnerable to various exploits and sometimes just to see if I should be using it. Most often this looks something like: find the software, download it, find the install and configuration documents, walk through them, find an error or two (documentation always seems to be out of date), fiddle with dependencies (database settings, etc.) finally get it mostly working (I think) and then test it out. I'm always wondering in the back of my mind if I've done it properly or if I've missed something, especially when it comes to performance issues (is the software slow, or did I mess up the database settings and not give it enough buffers?).

But it seems that finally some places are taking note of this and making their software available in a VM, fully configured and working, no fuss or mess, just download, run it, and play with the software. Personally I love this, especially for large and complex systems that require external components such as a database server, message queues and so on. No more worries about configuring all the add on bits, or making sure you have a compatible version and so on.

This really dovetails nicely with an increasing reliance on cloud computing, instead of buying a software package and having to maintain it you can buy a VM image with the software, essentially getting SaaS levels of configuration and support but still giving you the IaaS levels of control (you can run it in house, behind a VPN, you can configure it specially for your needs if you have to, etc.). The expertise needed to properly configure the database (which varies between databases hugely, and depending on what the product needs, i.e. latency? memory? bulk transfers? special character encoding?) is provided by the vendor who (at least in theory) knows best. I also think vendors will start to appreciate this, the tech support and time needed to guide customers through install and configuration and integration with other services and components can be replaced by "Ok sir I need you to upload the image to your VMware server (or EC2 or whatever) and turn it on... Ok now login as admin and change the password. Ok we're done."

Security is also a lot easier. The vendor can test patches knowing with certainty that they will work on customer's systems since the customer has the exact same system as the vendor, and customers can apply patches with a higher degree of confidence, knowing that they were tested in the same environment.

Reddit ships code as fully functional VM. - http://blog.reddit.com/2010/05/admins-never-do-what-you-want-now-it-is.html

Update: CERT release fuzzing framework, as a VMware image - http://threatpost.com/en_us/blogs/cert-releases-basic-fuzzing-framework-052710

http://www.cert.org/download/bff/