This is a bit wordy, so please bear with me. :)
We have a RAC cluster of 3 Oracle 11gR2 servers and purchased FusionIO ioDrive2 PCIx cards for these servers to use as OracleDB's Flash Cache. These servers run both our production db instance and a smaller development db. We would like to configure both instances to use these devices as their Flash Cache, dev db having a much smaller size.
It is my understanding that only 1 device can be configured as a destination for Flash Cache.
At the same time this ioDrive2 device supports what's called "Virtual Controllers". It basically splits this device into 2 to double the IOPs (total combined bandwidth remains the same).
So to put this whole picture together: I'll need 2 partitions, one for each db instance. If I utilize "Virtual Controllers" I'll have 2 equally-sized drives.
What I'm thinking of doing is to combine 2 virtual controllers into 1 using LVM and then partition that one combined volume for each db.
How much performance and management overhead will I get with LVM? Are there any caveats I should be aware of with this setup? Is there a better approach to this situation?
I'm doing something similar with another product, but what you're looking to do is actually pretty low-overhead. The devices I'm using present four 'drives' per PCIe card, and using MDRAID to convert that into a RAID0 device upon which I layer LVM has worked extremely well for me. You lose some blocks to metadata, but the I/O code-paths in the kernel are really well optimized for this.
The caveat here is that "RAID0" thing. I have had one of those virtual 'drives' fail on me, and that did indeed lose me the whole card (that's OK, this service is designed to tolerate whole-node failures). I don't know Oracle's flash-cache, so I can't tell you if it's tolerant of such failures.
I don't think anyone here could tell you how much performance overhead you'll experience. It really is a test and observe situation. I'll assume that all-else-equal, there would be a performance gain.
Do you have any specifications on the host system? When I was evaluating FusionIO and the need to (software) RAID 1+0 across multiple FusionIO Duo cards, I paid attention to the CPU, RAM and system board capabilities of the host server(s). There's going to be an impact CPU-wise if you're pushing the I/O that you seem to planning for.
Another question... What problem will this solve? Are you really just trying to add another tier of storage to the DB? For the RAC cluster, what is the working set of data; e.g. what drive capacities are you working with?
Do you already have everything in place and purchased?