I have a Dell NX3000- which is a Windows Server 2008 based unit. It has an additional cabinet full of drives attached, as well. I have spare licenses for Windows Server 2012 Standard that could be applied to this unit to turn it into a smb 3.0 server. Obviously, this is not a supported configuration from Dell but are there any known issues on this hardware series that would prevent this?
Absolutely. The NX3000 is essentially just a "rebranded" PowerEdge R710. You can use the process below to rebrand a system back to PowerEdge if you'd like, as this will ensure full OS compatibility. It's really not much more than a "switch" that Storage Server installations check for before deploying, so your Server 2012 install may be unaffected by it. I would still recommend rebranding beforehand just to be safe though.
BIOS.EXE /brandname
BIOS.EXE /FORCETYPE
may be necessary in some instancesOnce rebranded, you can essentially treat the hardware like it's PowerEdge sibling, aside from calling for any kind of support (will still be handled by storage support for hardware issues). The hardware is quite literally identical at this point to an R710, aside from the front bezel and your service tag sticker.
I just used this method to go from a Dell NX200 to a Dell T310 and it worked perfectly except the /brandname didnt work - you need to use /forcetype and it boots as the new box - with no issues.