I manually changed /etc/gnome/defaults.list file to set a different application than Totem for playing video files. It is actually a specific wrapper script around Totem that sets a few things before playing the video, and I created a file /usr/share/applications/auto-totem.desktop for it. All video/...= lines in this file specify my auto-totem.desktop file, including video/3gpp.
However, this doesn't work for this file type (works for all other video files) - these files open in original Totem (without my wrapper script), moreover, I cannot change the default application using "Preferences" menu. Totem is shown as default application for opening these files and when I select another application and click "Set as default", it has no effect - the default reverts to Totem.
What is strange, when I set a default application in "Preferences", the correct line video/3gpp=auto-totem.desktop is actually written to the file ~/.config/mimeapps.list. Moreover, xdg-mime query default video/3gpp shows correctly the auto-totem.desktop file for my application and mimeopen file.3gp correctly opens the file in my application. Only Nautilus behaves strangely and forces opening these files in Totem. I thought that maybe something is cached for the session so I logged out & logged back in, but this did not help.
This is not specific to my application, when I tried to set another application as default for video/3gpp files (like mpv or vlc), the same happens - I cannot set anything other than Totem.
How can I fix this?
Answering to myself - this is not a full answer, because I don't know what actually happened and why, but deleting my
~/.config/mimeapps.listfile helped. That is, after deleting the file the default application forvideo/3gppfiles was still Totem (despite entry in/etc/gnome/defaults.list) - that is probably some bug? - but at least I was able to change it using "Preferences" menu. Of course I had to recreate a few associations that were in my~/.config/mimeapps.listfile that I made previously.