To get a computer to update its group memberships without rebooting the computer, you can purge kerberos tickets with the command klist -li 0x3e7 purge. A subsequent gpupdate or gpresult will reflect the new group memberships.
However, this does not seem to work on a domain controller. Why?
The tickets are purged successfully, but a subsequent gpresult /r does not reflect the changes.
A domain controller would fetch GPO directly from itself, over loopback SMB, and loopback SMB doesn't use Kerberos – it generally uses NTLM.
NTLM in msv_sspi has special handling for loopback connections; it appears that instead of doing challenge/response and then querying the directory for user details, the server process directly peeks at the security token of the client process. So the effective groups can only be updated by restarting the relevant process, which most likely means a reboot.