Is Puppet or Chef suitable for managing very basic server config in a multi tenant environment?

Yes, this is certainly possible. Deciding if you should do so or not is up to you though.

Regarding your queries:

1) fair enough. The traffic is ssl based, so certificate management is important. Also don't trust any 'facts' which the client supplies relating to its identity, as these can be altered by the client. You want to rely on the client's ssl certificate to provide the authentication of who the server is. To be honest if you're using things like hiera properly and avoiding huge hostname based if-blocks in your code (which you really should) you'll be fine.

2) It shouldn't be, assuming you keep it patched. Properly configured, there's only a small vector for the puppetmaster to be attacked by the clients. That said, the effects if it did get compromised are large, so take care to lock it down.

3) That's really a testing and deployment issue. If you have solid puppet code, it won't screw your files. It does take a little while to get that sorted, but for the basics (like you need) not long.

Try Ansible (ansible.cc). May be it is for you. There is no agent running on your clients. It is growing very fast.

Another very good alternative is Salt Stack.

Ansible and Salt are easy to understand, you can use them as a command line tool if you want, like distributed shell.

Is Puppet (or similar) a suitable technology for taking care of basic but critical mass changes?

Yes, it can be used this way. I use it this to support external clients systems.

I don't want any server to be able to see any config it shouldn't

If you are using puppet, you must not enable autosign then. Autosign permits hosts to automatically request a certificate. Your configuration and permissions will almost certainly be tied directly to the CN in the certificate. You don't want random computer coming online and being able to claim that they are actually the system with all the secret high security stuff.

If you are really paranoid you can adjust the puppets fileserver settings to create shares that only some systems can access. The fileserver access is based on the certificates.

I don't want Puppet to make any changes which it shouldn't, or revert any manual changes done on the server.

There are a couple different approaches to permitting local changes.

One method I frequently use is below. Basically if you pass an list to a source, then the puppet try each item in the list. So I add the first item in the list to point at a local file.

  file { '/etc/ssh/sshd_config':
    ensure => present,
    source => ["/etc/ssh/sshd_config_local",
               "puppet:///modules/ssh/$ssh_config_file"],
    ...
  }

Another option would be to make use of symlinks. If someone wants to use the puppet version they symlink to the puppet version of a file. If they want maintain their config locally, then they don't create a symlink.

  file { '/etc/ssh/sshd_config_puppet':
    ensure => present,
    source => "puppet:///modules/ssh/$ssh_config_file",
    ...
  }

The other possibility is to use augeas to make line-level changes instead of changing entire files. Be very conservative about what you change.

3> There is no automatic undo in Puppet or in any such tools. You have to write explicit code to undo. Additionally, you can research the Environments feature of puppet, have a Testing lab where new code is tested (could be VMs), and use code review.

I don't want Puppet to make any changes which it shouldn't, or revert any manual changes done on the server.

For config files that are created using Puppets File type this can be achieved by setting:

replace => false,

I use that to generate some config files the first time that an application is deployed to a server, but then any edits to that config file won't be overwritten by Puppet.

However this does go against the philosophy of Puppet to be an idempotent deploy script.

It may be better if you can, to have separate admin-editable files which are not managed by puppet which are included from the files that are managed by puppet.

Puppet works best for many servers with identical configuration. E.g. you write all the configuration of a shared web server provided by your company, then create N instances of that server. Afterwards doing changes on all instances at once (e.g. you find out that need to change AllowOverride for all apache virtual hosts) will be really easy. You'll also be able to store all the configuration information in a single place and have it under version control. In the perfect case you'll be able to handle a hardware failure by throwing away the broken host, replacing it with a new one, setting the same hostname and signing the needed certificate. Everything else could be done by Puppet.

But if you end up with almost no configuration sharing between two hosts, using puppet may be less productive than doing configuration manually. Also managing half of the server's configuration with puppet and the other half manually may not make much sense.

Summary: If you can create uniform and structured configuration for the hosts you're going to manage, Puppet is your best friend, but if you'll have to handle each service (host, virtual host, database) specially Puppet won't add much value.