Questions[cassandra](server)

I'm having a hard time restoring snapshot on Apache Cassandra (version 3.0.9). As far as I can say, I'm following the procedure described on datastax blog, along with several other ones (for instance : http://datascale.io/cloning-cassandra-clusters-fast-way/). Yet I may be missing something, and everytime I make a restore, data is missing.

Setup : 6 nodes cluster (1 DC, 3 racks with 2 nodes each) with a replication factor set to 3. Machines are hosted on AWS.

Backup procedure (on each node) :

1. nodetool snapshot mykeyspace
2. cqlsh -e 'DESCRIBE KEYSPACE mykeyspace' > /tmp/mykeyspace.cql
3. nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk '{print $NF ","}' | xargs > /tmp/tokens

I get the files generated by the nodetool snapshot command and backup them along with tokens and cql on S3.

Restore procedure (for each node unless it's specified) :

(after having created new VMs)

1. Download snapshots, tokens and keyspace
2. Stop service cassandra
3. Delete /var/lib/cassandra/commitlog/* and /var/lib/cassandra/system/
4. Insert tokens into cassandra.yaml
5. Start service cassandra
6. Restore mykeyspace from mykeyspace.cql on one node only
7. Wait for replication and stop service cassandra
8. Delete .db files in folder /var/lib/cassandra/data/mykeyspace/
9. For each table copy snapshots files (.db, .crc32, .txt) into /var/lib/cassandra/data/mykeyspace/$table/
10. Restart service cassandra
11. Run nodetool repair mykeyspace -full, one node at a time

Result :

There are always missing rows, approximately the same quantity for each table but never the same ones. I tried to "mix up" a bit the procedure, like restoring keyspace before tokens, running nodetool refresh before repair, but I meet the same issue everytime.

Since I'm not far from having a "good" restore, I think that I'm missing something pretty obvious. Analyzing logs didn't really help, as they don't show any error/failure messages.

Any help would be welcomed :) I can of course give more information if needed.

edit : no one ? I updated the question with cassandra version (3.0.9), which I forgot in the first place. I tried again to restore, but no luck. I don't have any more idea really :(

I have gone through the recommended architecture of the Cassandra node configuration! according to which the recommended hardware infrastructure for the node is to have

RAM: 16-32 GB,
storage: 500GB - 1TB and
a 64 bit CPU with 8 cores

datastax documentation says

"Maximum recommended capacity for Cassandra 1.2 and later is 3 to 5TB per node. "

I have heavy write system, say 10K records per second, initial data storage requirement is 72TB, and if i go with 1TB per node, i will have to have almost 80 nodes (keeping in mind overheads).. The aim is to lower node number by adding more data storage capacity to each node.

my question is
1. according to documentation, 16-32 GB of RAM will work fine with the 500-1TB load of data. so when i have to add more disk space, 3-5TB per node, will i have to enhance RAM and CPU too?
2. is there any correlation between storage size and RAM + CPU

I've got a one-node Cassandra cluster which is currently stuck in a major compaction process. After executing a nodetool compactit started the compaction and I see the tmp-sstable files with a size of 0 bytes. But that's all. No progress for hours.

I already stopped the Cassandra service, but after a restart it continued the compaction again without progress. A nodetool compactionstats shows 0,00% progress and a remaining time of six minutes. But I already waited 24 hours. In the meantime I stopped all readers and writers without seeing any difference.

The version I use is 2.1.2. I already tried with OpenJDK 7 and with Oracle's Java 7, but no difference.

The logs show no indication of OOM or any other exception or warning.

I'm not sure if it is of any help, but I currently migrate away from Cassandra as it doesn't fit my needs. So I delete a lot of data which has been migrated. There are many rows with several hundred up to a few thousand columns. But there are a handful rows with a million of columns. They now have a lot of tombstones in them due to the migration (deletion) process.

Any help in how to debug the issue are welcome.

Briefly: if I have 5 Tb of data and want to deploy this on 5 cassandra servers - does each machine need to have 5 Tb of disk space for data (not counting log space)? From the docs it sounds like at times cassandra will need 2x the data size - so 10Tb / server or 10Tb total in the array?

How much RAM should each machine have? Assume that the 5Tb is all in the same column space. I had been planning to max out the RAM on each machine but I'm not sure that's enough. Do I need an array of servers with a total of 5Tb of RAM?

Currently we run a 4-node Cassandra ring in each of two datacenters. We would like to rebuild them into a single 8-node ring. All else being equal we would really like to have consistent reads, so we currently run QUORUM reads and writes. However, if we lose a datacenter it appears that this would cause many or all requests to fail due to inability to meet the ConsistencyLevel. Since we plan to send requests to both datacenters, switching to LOCAL_QUORUM shouldn't be enough to guarantee consistency.

Cassandra appears to be sorely missing ConsistencyLevel settings that are measured against only available nodes.

What can be done to get maximum consistency without availability failures in this scenario, and what has to be traded-off to get it?