Luke404's questions

Is it possible to share the same datastore between multiple VMware clusters of the same version (both 6.5 or both 6.7)?

It is the norm to share the datastore between multiple hosts in the same cluster (and it is required for vSphere HA), but I'm talking about multiple, otherwise independent clusters each one with its own nodes and vCenter.

By "sharing" I mean having both clusters actively working on the datastore with virtual machines and possibly the datastore heartbeat for HA. The individual VMs will be in only one cluster inventory, but on the same datastore there could be a mix of VMs from the two (or possibly more?) clusters.

My question is deliberately generic, but if you need a concrete example think about two Essential Plus clusters, three nodes in each cluster, all of them connected via iSCSI to the same LUNs. Reason for the question is to avoid hard splitting the available SAN space in two parts, one for each cluster, without being able to "rebalance" storage if one LUN/datastore gets full white the other still has available space.

We are trying out ActiveDirectory integration on some linux systems with SSSD.

So far so good we joined the linux systems to the domain and we can log into linux systems with AD-defined users. Right now every AD user can log into every Linux system which has been integrated with SSSD.

How can I let user Foo log into LinuxServer01 but prevent him to log into LinuxServer02? And/or how can I block a user to log into every linux systems and just let him on some specific ones?

When creating new Linux virtual machines using recent (>= 2014) distributions in VMware ESXi 5.5, should they mount their filesystems with the discard option ?

I make the assumption that the VM's admin does not necessarily know what is the actual underlying storage and that the VM is using whatever is the recommended "virtual controller" and I hope/suspect that if the VM sends out TRIM commands they could be useful with thin virtual disks (even if they are on non-SSD storage) or with thick virtual disks on SSD storage.

Does the discard mount option actually do something when used in an ESXi VM (eg. even if the virtual controller is a SCSI one and not a SATA one) ?

Does vSphere >= 5.5 do anything when the VM uses such discard option?

We just got a bunch of new Dell servers with current vSphere & vCenter versions (5.5, with possible future upgrade to 5.5U1). We'd like to add our older Dell R300 servers (Xeon 33xx series and onboard SAS6/iR controller, currently running ESXi 5.1 and still under support contract) to the same vCenter but Dell and VMware say they only support up to vSphere 5.1U2.

Now we need to choose between trying to run everything on 5.5 (if it runs on the R300s) or downgrade newer gear to 5.1 (we can still do that since the newer machines are not yet in production) or run a vCenter 5.5 with some 5.5 nodes and some 5.1 nodes until we refresh the older hardware (will probably be in the 12~18 months range).

Is vSphere 5.5 actually incompatibile with R300s (i.e., won't run) or is it just not covered by "official support" because they're older servers but runs anyway?

Sometimes when you connect to a multipath'ed iSCSI target you will have access to only some of its portals. This is common when the initiator is directly connected to the target's ethernet ports, as opposed to going though switches.

[my actual example at hand of such an infrastructure is a Dell MD3200i / MD3220i SAN with 4 directly connected servers]

When you do iSCSI discovery the initiator will get a list of all portals, even those that it is not actually connected to and can't speak to. With a dual controller, four ports per controller, you would see something like:

# iscsiadm --mode discovery --type sendtargets --portal 192.168.130.101
192.168.130.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234
192.168.131.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234
192.168.132.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234
192.168.133.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234
192.168.130.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234
192.168.131.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234
192.168.132.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.690b11C0123456789012345678901234
192.168.133.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.690b11c0123456789012345678901234

...but the host is physically cabled to the first (192.168.130.101) and sixth (192.168.131.102) ports, so it will never be able to speak with the other six portals.

Following typical documentation for iSCSI targets one ends up with all portals "known" by the initiator but the latter will only do the actual login on the ones of interest (two of them in the above example).

Should the "unreachable" ones be deleted from the initiator's configuration? Can they pose any problem being "known" even if not actually logged into ?

When an ESXi host is not under memory pressure (ie, running VMs are consuming less memory than physically available) how is "free" memory managed?

Does it have some sort of block cache for block devices (virtual disks)? anything else?

If a managed switch and all attached devices both support VLANs and Jumbo Frames, is it possible to mix VLANs with jumbo frames and VLANs with standard frames on the same switch port / ethernet card?

If it is possible, should I expect it to work in any environment or there are specific items (server ethernet card, ethernet card driver, server operating system, switch brand/model, etc.) that could cause problems in such a mixed environment?

I have some data whose source-of-truth is in a MySQL database, size is expected to max out at the some-thousands-rows range (in a worst-case scenario) and I'd like to use puppet to configure files on some servers with that data (mostly iterating through those rows in a template).

I'm currently using Puppet 3.0.x, and I cannot change the fact that MySQL will be the authoritative source for that data. Please note, data comes from external sources and not from puppet or from the managed nodes.

What possible approaches are there? Which one would you recommend?

Would External Node Classifiers be useful here?

My "last resort" would be regularly dumping the table to a YAML file and reading that through Hiera to a Puppet template, or to directly dump the table in one or more pre-formatted text file(s) ready to be copied to the nodes.

There is an unanswered question on SF about system users but the fundamental issue is probably similar to mine - he's trying to get data out of MySQL.

The NTP deamon if often used in its simplest mode, which is client/server: you specify one or more server directives in your ntp.conf and your clients will use those servers.

In addition to that, when you run your own NTP servers, it is good practice to peer them together, so if one of them looses connectivity to its upstream servers, it will get time from its peers.

But NTPd can also work with broadcast and/or multicast distribution of time data, with the documentation stating:

broadcast and multicast modes are intended for configurations involving one or a few servers and a possibly very large client population

The documentation also says elsewhere:

It is possible and frequently useful to configure a host as both broadcast client and broadcast server. A number of hosts configured this way and sharing a common broadcast address will automatically organize themselves in an optimum configuration based on stratum and synchronization distance.

I can see one obvious administrative benefit: you don't have to manually specify and update your list of NTP servers in the clients ntp.conf, so to me it looks tempting to use broadcast mode even for a small client population (say 5+ clients with 3~4 servers). I expect network traffic to be a little higher with broadcasts instead of client/server associations, but given the usual gigabit ethernet LAN the impact should be negligible unless you have a very very large number of hosts in the same broadcast domain.

At the end of the day, when should broadcast mode be used or avoided? Are there pros and cons I haven't seen?

I'm using Ubuntu 12.04 LTS server edition and I am modifying /etc/udev/rules.d/70-persistent-net.rules to define my own mappings of ethernet interfaces to MAC addresses; that file is initially generated by rules in /lib/udev/rules.d/75-persistent-net-generator.rules at system installation time (or at the first boot, I actually don't know and it doesn't matter here).

How can I be sure that my edited version will never ever be overwritten by anything?

Removing the persistent-net-generator, as suggested on some websites, is not the Right Thing™ to do as told by comments in the file itself: it will be overwritten by any update of the udev package. I'm looking for a more formally correct way to disable it.

Is it enough to just make sure that /etc/udev/rules.d/70-persistent-net.rules does exist? Maybe there are other events that could trigger its regeneration? (eg. adding or removing ethernet interfaces to the system?)

We all know (good example on SF) that Puppet hostname detection could be... fun.

At our company (and I guess we're not alone at this) we usually pre-configure servers at our offices and test them before bringing the gear to a remote datacenter and rack them. Of course the reverse dns will change when doing that, even if we don't change the actual hostname of the system.

We're slowly drafting our puppet setup and I'd like to be sure those moves won't create problems. My idea is to explicitly configure the desired full FQDN of the system as certname in puppet.conf at server provision time (before the very first puppet run). My process would look something like this:

1. basic o.s. installation
2. basic network configuration, enough to reach the internet and resolve dns
3. install puppet and set up certname
4. start puppet and let him manage the whole configuration
5. test, fix problems in config (via puppet), re-test, and so on...
6. manually stop puppet
7. set up new network configuration for the datacenter network
8. move the machine to DC
9. turn it on
10. puppet should automatically start and keep on doing its job

The process is supported by detecting the environment in puppet's manifests (eg. based on subnet, like they do at Wikimedia) and modify configuration as needed (eg. resolv.conf contents appropriate for each network). Each node's certname will never change for the whole system life cycle.

Is there any problem with this approach? Could it be improved?

I'm learning my way through configuration management in general and using puppet to implement it in particular. I have already done some generic research (also on SF) and right now I'm considering Apache VirtualHosts.

We host a lot of LAMP websites (it's currently in the hundreds range) on two systems: an Apache2/mod_php one and a MySQL one - basically the opposite of another question already on SF where he manages lots of servers with few vhosts each (if not actually one, I don't know). I have not yet put together a working config in puppet but it shouldn't be a problem, there are many examples and recipes out there.

In addition to the obvious apache configuration file(s) (no problem here I guess) every vhost would need to have some directories created and permissions checked (eg. a root dir for each vhost containing a documentroot, a dedicated tmp dir, a dedicated php session files dir, possibly SSL certificates, and so on) on the webserver, and a user + one or more databases on the MySQL server.

Adding a new vhost would require puppet to create those, removing one would require puppet to run some script that will backup user data and then remove the live data from the two servers, but also each and every puppet agent run would then check the existence of the directories, the db, permissions, etc.

Am I asking for trouble when going up to hundreds of virtualhost with all those checks running at every puppet run, especially the filesystem ones (on the webserver), and especially when in the future the systems will be loaded up more? (let's say we target the 1000~2000 websites range as a reasonable per-server maximum).

Is there any experience in doing that out there on the net? I googled but found nothing, also because there is a low signal/noise ratio when searching for "puppet" and "apache"...

I'm learning my way through configuration management in general and using puppet to implement it in particular, and I'm wondering what aspects of a system, if any, should not be managed with puppet?

As an example we usually take for granted that hostnames are already set up before lending the system to puppet's management. Basic IP connectivity, at least on the network used to reach the puppetmaster, has to be working. Using puppet to automatically create dns zone files is tempting, but DNS reverse pointers ought to be already in place before starting up the thing or certificates are going to be funny.

So should I leave out IP configuration from puppet? Or should I set it up prior to starting puppet for the first time but manage ip addresses with puppet nonetheless? What about systems with multiple IPs (eg. for WAN, LAN and SAN)?

What about IPMI? You can configure most, if not all, of it with ipmitool, saving you from getting console access (physical, serial-over-lan, remote KVM, whatever) so it could be automated with puppet. But re-checking its state at every puppet agent run doesn't sound cool to me, and basic lights out access to the system is something I'd like to have before doing anything else.

Another whole story is about installing updates. I'm not going in this specific point, there are already many questions on SF and many different philosophies between different sysadmins. Myself, I decided to not let puppet update things (eg. only ensure => installed) and do updates manually as we are already used to, leaving the automation of this task to a later day when we are more confident with puppet (eg. by adding MCollective to the mix).

Those were just a couple of examples I got right now on my mind. Is there any aspect of the system that should be left out of reach from puppet? Or, said another way, where is the line between what should be set up at provisioning time and "statically" configured in the system, and what is handled through centralized configuration management?

We currently have 4 Linux nodes with local storage, arranged in two active/passive pairs with storage mirrored using DRBD, running virtual machines (actually using Xen Hypervisor) for typical hosting workloads (mail, web, a couple VPS, etc.). We're approaching the (presumed) maximum IOPS of those servers, and we're planning to migrate to an external storage solution with two active nodes, with capacity for up to four active nodes.

Since we're an all-Dell shop I've done some research and found the MD3200 / MD3200i products should be the ones we're looking for. We are pretty sure we won't be attaching more than 4 hosts on a single storage and I'm wondering if there is any clear advantage for one or the other.

In theory I should be able to attach 4 SAS hosts to a single MD3200 (single links on a single controller MD3200, or dual redundant SAS links from each host to a dual-controller MD3200), or 4 iSCSI hosts to a single MD3200i (directly on its 4 GigE ports without any switch, again with dual links for the dual controller option).

Both setups should let us implement live VM migration since all hosts can access all the LUNs at the same time, and also some shared filesystem like GFS2 or OCFS2. Also, both setups should allow full redundancy of the whole system (assuming dual controllers in the storage).

One difference I can see is that the DAS solution is actually limited to 4 hosts while the iSCSI one should be able to grow to more hosts (adding two GigE switches to the mix).

One point for the iSCSI solution is that it would allow us to start out with our current nodes and upgrade them at a later time (we can't add other SAS controllers, but they already have 4 GigE ports each).

With the right (iSCSI|SAS) controllers I should be able to connect diskless nodes and boot them off the external storage which I think is a good thing (get rid of any local storage).

On the other hand, I would have thought the SAS one to be cheaper but it seems like an MD3200 actually costs a little less than an MD3200i (?)

(please note: I've used Dell gear in my examples since that's what we're looking for but I assume the same goes with other vendors)

I would like to know if my assumptions above are correct, and if I'm missing any important difference between the two setups.

I'm evaluating the possibility to use two off the shelf servers to build a cheap iSCSI redundant SAN. The idea is to run linux, pacemaker, and an iSCSI target - something like the SAN Active-Passive on linux-ha-examples.

The same page scares me a little when I read:

During the switchover of the iscsi-target one can detect a gap in the protocol of write-test.log. In our setup we observed a delay of 30s. There are problems reported in connection of ext3 and an iscsi failover This configuration has been tested with ext2 and ext3 and worked with both filesystems.

Has anyone put in production a redundant iSCSI SAN made out of linux boxes? Is a failover event really that bad? A 30 seconds freeze in I/O sounds like a disaster to me, isn't it?

We currently have an active-active 2 node cluster running virtual machines. Every node has two disks, every disk is DRBD mirrored on the other node. Every node runs virtual machines out of his primary drbd device, and the pacemaker cluster will handle failover (if a node fails, the other becomes primary on both drbd devices and run all the VMs). This is colocated in a datacenter so our costs (beside hardware acquisition) are driven by how many rack units we occupy.

When you start out small this is a great solution, it fits in 2U of rack space (assume ethernet switch(es) is/are already there) and it's 100% redundant. But it's also a slightly difficult setup to manage and it suffers when I/O load goes too high (I guess that's just because of the low number of spindles).

I'm wondering what could be the best solution to scale above our hardware capacity while still being cost-effective and be as redundant as it's reasonable:

• go on by adding more two node clusters with internal storage, maybe with bigger hardware (eg. 2U servers with more disks)
• still use two nodes clusters but with external direct attached storage (stuff like 1U or 2U disk enclosures with SAS links) - see note below
• separate storage and VMs (eg. a pair of storage nodes, mirrored by DRBD, which export iSCSI and manage failover by moving the iSCSI target IP, coupled with two or more diskless nodes which run VMs out of the static iSCSI target IP) - this seems to be what others are doing?
• use something different from standard servers for the storage part (dedicated storage solutions on gigabit ethernet?)
• anything else altogether ?

The split storage/application servers seems the most flexible and reasonable solution to me, we could easily add more storage nodes when needed and still use the current application servers or do the other way around when we hit capacity limits.

What do you think are good / bad choices? Do you already have experience on this kind of stuff without bigger budgets (I tend to rule out fibre channel or 10000 euros storage appliances)?

EDIT: To be clear, the idea is that by leveraging modern (and free) software you could implement redundancy just by adding more "commodity" hardware. It's not going to be screamin' fast nor super high availability, but it will let our VM run even if a motherboard dies for as long as it takes to get a spare to the DC and replace the part.

EDIT: I removed the usb mention because it's really not going anywhere (and thanks for pointing that out in replies). I don't really know how I forgot about SAS enclosures. As an example from the Dell website, an MD1000 is 2U with SAS links. Two enclosures attached to two storage nodes via SAS and they could do redundancy and export iSCSI.

There are many black lists out there, with bad and good reputations. Also, some people recommend to use RBL lists only in score-based systems (like spamassassin) while others (including commercial service providers, think Barracuda) extensively use lists to hard block messages before even looking at that.

The problem in using an RBL list to directly discard messages come from false positives, that 99% of the times would otherwise survive a score based system. I'm wondering if there are lists out there that could be put at good use in pre-queue, that have very very low false positives even if they maybe are less effective at catching true spam messages. Such lists would be good, in combination with greylisting, to skim a good part of the mail intake before running it through a content analysis system.

We are currently using only Abuseat's CBL to block SMTP connections, and we're not aware of any false positive introduced by this. Lists from the Spamhaus Project also look promising, but which ones? or all of them (like using ZEN)?

• Do you use (trust) RBL lists to directly block SMTP sources?
• Is there any known study on false positives rate or, do you have any experience to report?
• What lists would you recommend for this kind of filtering?

I am no LXC expert, but as far as I know it is really similar to linux-vserver.

If that is correct, I wonder why there is another player in the already crowded virtualization camp? What does LXC provide (or promise) that linux-vserver doesn't have?

I'm a long time linux-vserver user, I remember it was around many years ago before all this virtualization hype. Development slowed down in recent times, but never totally stopped - I know because developers helped me to bring it back to a useful state on linux/hppa (in Gentoo, at least). They regularly push out patches against the latest kernels, using all the latest goodies (the same container infrastructure used by LXC) and even integrating with GRsecurity.

People who started the LXC effort probably had a good reason to do so instead of joining the already well established linux-vserver base? Then I'd like to know.

Is it all about mainline integration? Then why not adapt linux-vserver for integration, like Xen people are doing with XenLinux?

I am collecting SNMP data on-site for a number of systems, currently using Cacti. These systems are spread on a number of sites that aren't always connected to internet, but I also need to centralize the data on a single system (datacenter housed server) and get graphs out of it.

If I directly poll remote systems with a centralized Cacti I'd lose data when a site is not connected to internet. I should record data on-site (I have a server at each site and I can run whatever I want on it) and then 'sync' everything to the central system.

One hack could be a cacti or directly an rrdtool on site and then periodically rsync RRD data to the central Cacti system, but that doesn't sound like a 'clean' solution: every RRD would have to be defined at both places and rsync scripts setup with the specific file names.

Can you suggest a better solution? Cacti is not a requirement but I'd like to use something like that on the central system. On-site systems need only to collect data I don't need to graph it there or manage users rights to view data and stuff like that, users will only access the centralized system.

Given the very specific nature of the subject (we're not talking about mailboxes, just the spool; we're not talking about other filesystems, just ext3; and so on...) and the maturity of the softwares involved (linux kernel, ext3fs, postfix) I'd think there should be a more or less agreed on set of best practices to filesystem related tuning.

I'm trying to get a roundup of them:

• data=journal became the default in recent kernels (somewhere around 2.6.30 IIRC) so we should be ok with that
• Wietse Venema says atime must be on, but Postfix documentation recommendsnoatime while talking about the Incoming Queue. Does that mean that postfix needs atime on just for some queue directories and will benefit from noatime on the others? can we use noatime if we just don't use ETRN?
• filesystem can be mounted nodev,noexec,nosuid - no* won't prevent you from setting attributes (postfix uses exec attr) they just won't have any effect (we don't run anything from the spool)
• the fsync() issue cited by Wietse and/or the chattr -S are probably linked to sync/async options of ext3fs but I do not understand them enough. Mouting the filesystem with async option is equivalent to chattr -R -S the whole fs? Seems like it will increase performance, but will that pose a risk of "loss of mail after a system crash" or is it really "safe on /var/spool/postfix" ?
• would you tune anything else on postfix-2.6.x to work better on ext3 or do you leave defaults everywhere?
• is there a "best" linux I/O scheduler for this kind of workload (namely CFQ or deadline?) or that's something that will vary too much based on hardware configuration?
• would you tune anything else in the filesystem or in the kernel?
• anything else?

References:

• Postfix Performance here on SF
• Postfix documentation about the Incoming Queue
• Wietse Venema in Best file system on [email protected] here
• Postfix and ext3 on [email protected] here and there