“1. So what does the 5th level modifier do?”

It chooses the fifth level. The 2nd level modifier (just regular shift) chooses the second level, and the 3rd level chooses the third level.

The reason that the 5th level modifier chooses the fifth level instead of the fourth level (right after the 3rd level modifier) is that these three modifiers can be combined in different ways to choose eight different level (this includes not using any of the modifiers, which is just pressing a key without any such modifiers). There are eight combinations since 2↑3 = 8. For the first two modifiers there are four combinations:

• No modifiers (level 1)
• 2nd level modifier, a.k.a. shift (level 2)
• 3rd level modifier (level 3)
• 2nd + 3rd modifier (level 4)

Adding the third modifier (5th level) adds four more levels.

“2. How do I set it up so that I can get 5 or six different characters out of a single key (modified by Shift, Alt Gr and Right Ctrl for example)?”

First of all you need a layout which uses five or more characters per key. You can choose an existing layout, but if you simply want to add more levels to the layout you are currently using, that might not be optimal. For example, there is the Neo keyboard layout (†1), but that is “optimized for the German Language” so not that great for most English speakers.

Still, it might be instructive to have a look at what keys Neo has chosen for the level 5 modifier.

Neo apparently calls the level 5 modifier for “Mod4”. So as you can see it uses the key to the right of the left shift key (LSGT) and the right alt key/AltGr key. This is on a European keyboard (105 keys) which compared to the US keyboard (104 keys) has a narrow left shift and tall and thin return key. For a keyboard layout which is based on the US keyboard one might have to—or prefer to—choose other keys.

Choosing a level 5 modifier key

In sup’s answer he chooses to use Xmodmap. Nowadays tools like Setxkbmap is recommended over Xmodmap. Xkb comes with many predefined options for things like choosing the level 5 modifier. If you consult man xkeyboard-config and search for lv5 you’ll find these options:

┌──────────────────────┬───────────────────────────────────────────────────┐
│ Option               │ Description                                       │
├──────────────────────┼───────────────────────────────────────────────────┤
│ lv5:lsgt_switch_lock │ <Less/Greater> chooses 5th level; acts as onetime │
│                      │ lock when pressed together with another 5th level │
│                      │ chooser                                           │
├──────────────────────┼───────────────────────────────────────────────────┤
│ lv5:ralt_switch_lock │ Right Alt chooses 5th level; acts as onetime lock │
│                      │ when pressed together with another 5th level      │
│                      │ chooser                                           │
├──────────────────────┼───────────────────────────────────────────────────┤
│ lv5:lwin_switch_lock │ Left Win chooses 5th level; acts as onetime lock  │
│                      │ when pressed together with another 5th level      │
│                      │ chooser                                           │
├──────────────────────┼───────────────────────────────────────────────────┤
│ lv5:rwin_switch_lock │ Right Win chooses 5th level; acts as onetime lock │
│                      │ when pressed together with another 5th level      │
│                      │ chooser                                           │
└──────────────────────┴───────────────────────────────────────────────────┘

So the options that are documented are the LSGT key, the right alt key, the right Windows key, and the left Windows key. As we’ve seen, Neo uses the first two options.

Say you want to use the lv5:ralt_switch_lock option with some layout. This command sets up the us layout with this level 5 modifier key:

setxkbmap -option "lv5:ralt_switch_lock" us

So for any of the options which are documented in man xkeyboard-config, use that as an argument to -option when using Setxkbmap.

In case these options are not enough and you want to define another key to be the level 5 modifier, it is instructive to look at the declarations for these options. They seem to be in /usr/share/X11/xkb/symbols/level5. The declaration for lv5:ralt_switch_lock looks like this:

partial modifier_keys
xkb_symbols "ralt_switch_lock" {
  include "level5(lock)"
  key <RALT> {
    type[Group1] = "EIGHT_LEVEL",
    symbols[Group1] = [ ISO_Level5_Shift, ISO_Level5_Shift, ISO_Level5_Shift, ISO_Level5_Shift,
                        ISO_Level5_Lock, ISO_Level5_Lock, ISO_Level5_Lock, ISO_Level5_Lock ]
  };
};

The right side of symbols[Group1] declares what key or character each level of the key should produce. This uses all the eight possible levels since it is supposed to lock the level 5 modifier when pressed together with another level 5 shift key. So levels 5–8 emits ISO_Level5_Lock since all of these levels are accessed by holding down the level 5 shift and some combination of the other shift keys (none to all of them). If you only want the shift behavior and not also the lock-if-two-pressed behavior, you can make a simpler declaration (from the same file):

// The right Alt key (while pressed) chooses the fifth shift level.
partial modifier_keys
xkb_symbols "ralt_switch" {
  key <RALT> {
    type[Group1]="ONE_LEVEL",
    symbols[Group1] = [ ISO_Level5_Shift ]
  };
  include "level5(modifier_mapping)"
};

Adding extra levels to a layout

Xkb layouts are pretty modular. You can include existing layouts and alter them. So we can choose an existing layout with less than 5 levels and add some levels to it.

You can make new layouts by copying the file with your layouts to /usr/share/X11/xkb/symbols/, or wherever it is that your Xkb tools look for layouts. But this requires sudo privilege so it is not the best solution. Alternatively you can copy the directory structure of xkb/ and put your layouts in the appropriate directory under xkb/, namely symbols/. So say you have made a file xkb-tree/symbols/my-layouts. In order to use layouts defined in this file, you can use the -print option of setxkbmap and then pipe the output to xkbcomp. This wrapper script take the layouts as input:

#!/bin/sh

# `-option ""` resets all options.
setxkbmap -option "" -layout "$1" \
    -print |
    xkbcomp -I"<path-to-project>/xkb-tree" - "$DISPLAY"

As far as the file with the layout is concerned, let’s say that we want to expand the “US international” layout, which uses four levels. First of all we set up the level 3 and level 5 shift keys in the same way that Neo does:

partial alphanumeric_keys
xkb_symbols "mine" {
    // US international
    include "us(intl)"
    // Set up level 3 modifiers.
    include "level3(modifier_mapping)"
    include "level3(bksl_switch)"
    include "level3(caps_switch)"
    // Set up level 5 modifiers.
    include "level5(modifier_mapping)"
    include "level5(lsgt_switch)"
    include "level5(ralt_switch)"
};

Now we want to add some extra characters. Let’s just use the first four keys of Qwerty layouts as an example. These keys are Q to R, which correspond to <AD01> to <AD04>. First we need to make the keys capable of using up to eight levels:

key.type[Group1] = "EIGHT_LEVEL";

For the first four levels, in order to not redefine them, we map them to NoSymbol. Then we add two levels. On level 5 we add the arrows ←↓↑→ (†2). On level 6 we add the direction keys. Our layout ends up looking like this:

partial alphanumeric_keys
xkb_symbols "mine" {
    // US international
    include "us(intl)"
    // Set up level 3 modifiers.
    include "level3(modifier_mapping)"
    include "level3(bksl_switch)"
    include "level3(caps_switch)"
    // Set up level 5 modifiers.
    include "level5(modifier_mapping)"
    include "level5(lsgt_switch)"
    include "level5(ralt_switch)"

    // Make the following keys capable of using eight levels.
    key.type[Group1] = "EIGHT_LEVEL";

    key <AD01> { [
        NoSymbol, NoSymbol, NoSymbol, NoSymbol,
        leftarrow, Left
    ] };
    key <AD02> { [
        NoSymbol, NoSymbol, NoSymbol, NoSymbol,
        downarrow, Down
    ] };
    key <AD03> { [
        NoSymbol, NoSymbol, NoSymbol, NoSymbol,
        uparrow, Up
    ] };
    key <AD04> { [
        NoSymbol, NoSymbol, NoSymbol, NoSymbol,
        rightarrow, Right
    ] };

};

In order to use it we can use the wrapper script that we made:

./setxkbmap-wrapper "my-layouts(mine)"

†1: Neo is de(neo) in Xkb; setxkbmap "de(neo)".

†2: In order to find the symbolic names that Xkb uses, search for a file named keysymdef.h. It should be under a directory named X11/. You can also use Unicode names, for example U2192 for U+2192.

Well, after a bit of digging:

(1) Key to choose 5th level does some weird stuff (regarding some French layouts) that is not relevant to what I wanted to do. Basically it sets which key is going to act as ISO_Level5_Shift. ISO_Level5_Shift is used in some obscure layouts. This answer is based on the number of results google gives when searching for "ISO_Level5_Shift" and on these links.

(2) It cannot be used to get 6 symbols out of one physical key. One way to achieve this is with the use of file .xmodmap placed in $HOME directory.

First, command

xmodmap -pm

gives what keysymbols act as various modifiers. On my system it was like follows:

xmodmap:  up to 4 keys per modifier, (keycodes in parentheses):

shift       Shift_L (0x32),  Shift_R (0x3e)
lock        Caps_Lock (0x9)
control     Control_L (0x25),  Control_R (0x69)
mod1        Alt_L (0x40),  Meta_L (0xcd)
mod2        Num_Lock (0x4d)
mod3        
mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce),  Hyper_L (0xcf)
mod5        ISO_Level3_Shift (0x5c)

mod3 needs to set up in order to use 6 symbols with one key. This needs to be placed into .xmodmap file:

clear mod3
clear mod5
keycode 94 = Mode_switch
add mod3 = Mode_switch
add mod5 = ISO_Level3_Shift

Then xmodmap -pm gave me:

shift       Shift_L (0x32),  Shift_R (0x3e)
lock        Caps_Lock (0x9)
control     Control_L (0x25),  Control_R (0x69)
mod1        Alt_L (0x40),  Meta_L (0xcd)
mod2        Num_Lock (0x4d)
mod3        Mode_switch (0x5e),  Mode_switch (0xcb)
mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce),  Hyper_L (0xcf)
mod5        ISO_Level3_Shift (0x5c),  ISO_Level3_Shift (0x6c)

Where keycode 94 is the key I chose to act as mod3. Keycodes of various physical keys can be found with the command line tool xev (just run it and press various keys, keycodes will show up).

Then, if I place into .xmodmap file

keycode 54 = 1 2 3 4 5 6

or run

xmodmap -e "keycode 54 = 1 2 3 4 5 6"

I get six different symbols out of one key with using altgr, shift and keycode 94 (it is a useless key next to left shift on my physical layout).

UPDATE: if you need just five symbols, it is handy to set it up like this:

keycode 54 = 1 2 3 3 5 6

It occasionaly happens to me that I get 4 instead of 3 otherwise.