My organization shelled out for a gTLD, and we currently use 3rd party (Verisign) to host it on their DNS servers. However, there are 2 issues: we pay quite a bit for that yearly (that's the lesser issue); updating data there is... convoluted (also for our organizational reasons).
However, we run quite huge DNS system for our other (non-TLD) domains. I have tried searching if it's possible to self-host gTLD, but while I found some info for new gTLDs, I haven't for transferring existing gTLDs.
Can anyone point me to some overview/howto/readme, or can just speak from experience?
EDIT: for sake of having a reference point, let's say I already own .example. gTLD. Currently it is maintained on Verisign DNS servers. I want to be able to maintain it on my own DNS servers.
Håkan is basically correct. The approval process for your gTLD included extensive technical testing of the platform serving it, both for conformance to DNS specifications (both RFCs and gTLD-particular specs from ICANN), world-wide availability over both IPv4 and IPv6, performance of registry interfaces (EPP, that is) and more. Since you're using Verisign you may not have noticed that this happened, since their platform hosts a whole bunch of TLDs and is known to be good (in that regard, at least). If you are to move your gTLD to an entirely new platform, I suspect that ICANN will want that the testing be repeated. I strongly recommend that you contact ICANN directly and ask how to handle this.
[Note: I know these things because I was involved in building the technical testing platform for the current set of new gTLDs, and personally wrote most of the DNS standards conformance tests for it.]
I don't have personal experience with the operation of a New gTLD but the answer surely must be that it is possible for you to operate your own nameservers (and other services). I think the question rather is "at what cost", to which I do not have a direct answer.
IANA's technical requirements for nameservers appears to be a straightforward technical baseline, essentially just normal best practices for any domain. (You should obviously still review them.)
Making changes to a gTLD delegation also appears relatively straightforward, IANA has information on how to manage a delegation readily available and a web-based management interface.
However, browsing through the New gTLD Applicant Guidebook, it would appear that during the application process you provided IANA with documentation of how you will operate in compliance with their policies, a good chunk of which appears to be about the technical operation of the gTLD registry and nameserver services (presumably answered, at least to some extent, based on your use of these externally provided services which you now want to replace).
It seems that it would be prudent to review your own application and the agreements you entered with IANA to assess the impact of these planned changes when it comes to complying with policy and any additional technical requirements mandated therein.