Home / server / Questions / 810968
Accepted
P. Bender
Asked: 2016-10-25 07:59:12 +0800 CST

Cassandra snapshot restoring : random missing data

  • 772

I'm having a hard time restoring snapshot on Apache Cassandra (version 3.0.9). As far as I can say, I'm following the procedure described on datastax blog, along with several other ones (for instance : http://datascale.io/cloning-cassandra-clusters-fast-way/). Yet I may be missing something, and everytime I make a restore, data is missing.

Setup : 6 nodes cluster (1 DC, 3 racks with 2 nodes each) with a replication factor set to 3. Machines are hosted on AWS.

Backup procedure (on each node) :

  1. nodetool snapshot mykeyspace
  2. cqlsh -e 'DESCRIBE KEYSPACE mykeyspace' > /tmp/mykeyspace.cql
  3. nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk '{print $NF ","}' | xargs > /tmp/tokens

I get the files generated by the nodetool snapshot command and backup them along with tokens and cql on S3.

Restore procedure (for each node unless it's specified) :

(after having created new VMs)

  1. Download snapshots, tokens and keyspace
  2. Stop service cassandra
  3. Delete /var/lib/cassandra/commitlog/* and /var/lib/cassandra/system/
  4. Insert tokens into cassandra.yaml
  5. Start service cassandra
  6. Restore mykeyspace from mykeyspace.cql on one node only
  7. Wait for replication and stop service cassandra
  8. Delete .db files in folder /var/lib/cassandra/data/mykeyspace/
  9. For each table copy snapshots files (.db, .crc32, .txt) into /var/lib/cassandra/data/mykeyspace/$table/
  10. Restart service cassandra
  11. Run nodetool repair mykeyspace -full, one node at a time

Result :

There are always missing rows, approximately the same quantity for each table but never the same ones. I tried to "mix up" a bit the procedure, like restoring keyspace before tokens, running nodetool refresh before repair, but I meet the same issue everytime.

Since I'm not far from having a "good" restore, I think that I'm missing something pretty obvious. Analyzing logs didn't really help, as they don't show any error/failure messages.

Any help would be welcomed :) I can of course give more information if needed.

edit : no one ? I updated the question with cassandra version (3.0.9), which I forgot in the first place. I tried again to restore, but no luck. I don't have any more idea really :(

backup restore data-loss database-backup cassandra

2 Answers

  1. The sed command in that blog post, which is supposed to append -Dcassandra.load_ring_state=false to the $JVM_OPTS, has no effect in its current form.

    If you were copying that command directly from the blog post, it's possible that's the issue. You could try this one instead which places it at the bottom of the file:

    sudo sed -i '$ a\JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"' /etc/cassandra/cassandra-env.sh

    Also you'll need to do a nodetool repair -pr <ks> on each node, one by one, after following this procedure.

    • 0
  2. Best Answer

    Ok, end of story, stupid me ! The initial_token line in cassandra.yaml was wrongly "seded" during my restore procedure. If there is no space after the ':' for the initial_token key, then cassandra fails to launch. therefore the line was kept commented and tokens not interpreted !

    tldr :

    • initial_token:<values> = WRONG
    • initial_token: <values> = GOOD

    Thanks a lot to you Josh Purvis for insisting on the high importance of this parameter :-)

    • 0