Get new locale for country which not exists? How?

Yes, since you installed in Estonia, Ubuntu 'guessed' that you prefer date and time formats, currency etc. 'the Estonian way'. If you want e.g. English instead, you can open Language Support, select the Regional Formats tab, and change it.

Edit:

If the weekday and month names are your primarily concern, while you want to keep other format aspects to Estonian as far as possible, one option is to open the ~/.profile file for editing and add this line:

export LC_TIME=en_DK.UTF-8

There are several ways. The easiest one, is to specify the exceptions. Let me show me my configuration using screenshots. What you will open is the "Languages Support" application. On the first run, it usually asks you to install the missing components. Let it do so.

Go to "Install Remove Languages" and get all the languages that you are interested in. For me that would be English, Dutch, French and German:

Now, in the main window, drag the languages you want for displaying on top. They are top to bottom, in preference. I prefer, in order: en_GB, en_US and finally just en:

Finally, go to the second tab called "Regional Formats". There you select the date/time/currency formats you want to use. For me that's fr_LU:

That's basically it. In my case, everything is displayed in English, but my formats are according to the Luxembourgish standard.

If you click "Apply System-Wide", it becomes a system-wide default. I usually do this, individual users can still change it to their liking (and even revert it for the system if they have an administrative account).

Alternative method: Set environment variables. Basically that's what the above application does for you. Here are mine:

$ env | grep -e LC_ -e LANG
LC_PAPER=fr_LU.UTF-8
LC_ADDRESS=fr_LU.UTF-8
LC_MONETARY=fr_LU.UTF-8
LC_NUMERIC=fr_LU.UTF-8
LC_TELEPHONE=fr_LU.UTF-8
LC_IDENTIFICATION=fr_LU.UTF-8
LANG=en_US.UTF-8
GDM_LANG=en_US
LC_MEASUREMENT=fr_LU.UTF-8
LANGUAGE=en_US
LC_TIME=fr_LU.UTF-8
LC_NAME=fr_LU.UTF-8

If you want to change them system wide, the file that you are interested in /etc/default/locale.

Finally, you can create a new locale. It's what I used to do, but the environment variable system works good enough for me. Basically, the locale definitions files can be found in /usr/share/i18n/locales. It contains a bunch of files named xx_XX. You pick the one that fits the formatting you want. For me that would be fr_LU. Now copy that to the (inexisting) locale you want to create: cp fr_LU en_LU. At this point edit the new file and replace the language name by the word English. In my case that was French into English. Finally, you need to add the new locale to /etc/locale.gen and and run sudo locale-gen and then use that in /etc/default/locale. Needless to say this is a quite complicated way, and I'm writing this from memory. I almost certainly forgot something.

For the locale generating (thanks to Gunnar for the wise additions), this should be the full procedure. I included output and commands.

jawtheshark@bruce:~$ sudo mkdir -p /usr/local/share/i18n/locales
jawtheshark@bruce:~$ sudo bash -c "sed s/French/English/ /usr/share/i18n/locales/fr_LU > /usr/local/share/i18n/locales/en_LU "
jawtheshark@bruce:~$ sudo bash -c "echo en_LU.UTF-8 UTF-8 >> /etc/locale.gen"
jawtheshark@bruce:~$ sudo locale-gen
Generating locales (this might take a while)...
  de_AT.UTF-8... done
  de_BE.UTF-8... done
  de_CH.UTF-8... done
  de_DE.UTF-8... done
  de_LI.UTF-8... done
  de_LU.UTF-8... done
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IN.UTF-8... done
  en_LU.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_US.UTF-8... done
  en_ZA.UTF-8... done
  en_ZM.UTF-8... done
  en_ZW.UTF-8... done
  fr_BE.UTF-8... done
  fr_CA.UTF-8... done
  fr_CH.UTF-8... done
  fr_FR.UTF-8... done
  fr_LU.UTF-8... done
  nl_AW.UTF-8... done
  nl_BE.UTF-8... done
  nl_NL.UTF-8... done
Generation complete.
jawtheshark@bruce:~$ sudo bash -c 'echo LANG="en_LU.UTF-8" > /etc/default/locale'

Sorry, for yet another edit. The above create locale file will still display dates in French. Since locale files define what is displayed, you can now edit that file (in my example /usr/local/share/i18n/localesen_LU). You will look for LC_TIME and END LC_TIME. Now, replace all the stuff within those two lines, with the words copy "en_GB". Regenerate your locale (sudo locale-gen) and the new locale is ready for use. There might be other linguistic remnants, and you'll have to adapt those too if you are irked by them.

However Doing all this, tells us it will be easier to do the following. Edit the /etc/defaults/locale file to reflect what you want. Above I pasted mine, it is full of fr_LU.UTF-8. Each of these variables represent one aspect of the way locale-specific data is represented. This means, to achieve the same thing we did by creating a new locale, we could simple do this (content of /etc/default/locale:

LANG=en_LU.UTF-8
LC_NUMERIC="fr_LU.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_MONETARY="fr_LU.UTF-8"
LC_PAPER="fr_LU.UTF-8"
LC_IDENTIFICATION="fr_LU.UTF-8"
LC_NAME="fr_LU.UTF-8"
LC_ADDRESS="fr_LU.UTF-8"
LC_TELEPHONE="fr_LU.UTF-8"
LC_MEASUREMENT="fr_LU.UTF-8"

This basically tells the system: format anything to the fr_LU.UTF-8 locale, except for LC_TIME, because that I want to see in en_GB.UTF-8. This should be the solution you are looking for.

Remark: You also might want to remove or change these environment variables in ~/.pam_environment. Those per-user and will override the default settings you set in /etc/default/locale. GUI applications, like "Language Support" might overwrite those-

1. Click the icon at the very right of the menu bar and select System Settings.
2. In the Personal section, click Language Support.
3. Click Install / Remove Languages.... The Installed Languages window lists all the available languages, with the currently installed languages checked.
4. Check the languages you want to install, and uncheck those currently installed languages you want to remove.
5. Click Apply Changes.
6. Administrative privileges are required. Enter your password, or the password for the requested administrator account.

For more see this link