Home / server / Questions / 527031
In Process
freedev
Asked: 2013-07-30 06:10:11 +0800 CST

mongorestore from stdin

  • 772

On a remote server I have a file which is the mongodump output compressed, let me say a file called mongodb.tar.gz.

Inside mongodb.tar.gz there is a directory structure like this:

dump/dbname/
dump/dbname/blogs.bson
dump/dbname/blogs.metadata.json
dump/dbname/editors_choice.bson
dump/dbname/editors_choice.metadata.json
...

Is there anyway to restore this dump without download and uncompress entire file locally?

I mean something like:

curl http://remoteserver/mongodb.tar.gz | gunzip | mongorestore -d dbname
mongodb

4 Answers

  1. You can only pipe compressed files which contain one collection.

    You could do:

    curl http://remoteserver/mongodb.collection.gz | gunzip -c | mongorestore -d dbname -c collectionname -

    The -c gunzip option is needed so it writes to stdout and the last -so mongorestore expects input from stdin.

    Tested with version 3.0.7 (doesn't work with v2.6.4).

    • 3

  2. At the moment, this is not possible, at least not without writing something yourself. The feature has been requested as SERVER-4345 and SERVER-5190 but there are several issues with an immediate implementation based on how the current tools work (i.e. it is not simple to do).

    • 2

  3. Although only a partial answer, you could use fuse to mount the .tar.gz file after downloading it.

    Seeking a direct answer to the other part, I asked question 730494.

    • 1

  4. Well I did it and it wasn't pretty. What I did was first extract only the metadatas from the tarball since they couldn't be directly piped into the mongorestore command which accepts only BSON.

    After extracting the metadata I ran two restores: first the normal mongorestore with the folder as a parameter to restore the metadata.

    Then in the second restore I read the file names of the BSON files from a file I had created earlier and for each file I untarred it to STDIN and piped the result to mongorestore. Yes it was messy but hey, it works!

    To see the abomination in its full glory here's the repo: https://github.com/datascienceproject2019-codescoop/codescoop-models

    And here's the script https://github.com/datascienceproject2019-codescoop/codescoop-models/blob/master/commands.sh

    The restore script is in a different file since piping to docker exec is too difficult: https://github.com/datascienceproject2019-codescoop/codescoop-models/blob/master/gh_mongo_scripts/restore.sh

    I used Mongo 4.0.6

    EDIT: Buuut it is a lot slower to use streams than to just read from the extracted files. So I probably did this all for nothing since extracting temporarily 26 GBs of extra files isn't that big of a deal. Oh well.

    • 0