Questions[data-loss](server)

I'm having a hard time restoring snapshot on Apache Cassandra (version 3.0.9). As far as I can say, I'm following the procedure described on datastax blog, along with several other ones (for instance : http://datascale.io/cloning-cassandra-clusters-fast-way/). Yet I may be missing something, and everytime I make a restore, data is missing.

Setup : 6 nodes cluster (1 DC, 3 racks with 2 nodes each) with a replication factor set to 3. Machines are hosted on AWS.

Backup procedure (on each node) :

1. nodetool snapshot mykeyspace
2. cqlsh -e 'DESCRIBE KEYSPACE mykeyspace' > /tmp/mykeyspace.cql
3. nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk '{print $NF ","}' | xargs > /tmp/tokens

I get the files generated by the nodetool snapshot command and backup them along with tokens and cql on S3.

Restore procedure (for each node unless it's specified) :

(after having created new VMs)

1. Download snapshots, tokens and keyspace
2. Stop service cassandra
3. Delete /var/lib/cassandra/commitlog/* and /var/lib/cassandra/system/
4. Insert tokens into cassandra.yaml
5. Start service cassandra
6. Restore mykeyspace from mykeyspace.cql on one node only
7. Wait for replication and stop service cassandra
8. Delete .db files in folder /var/lib/cassandra/data/mykeyspace/
9. For each table copy snapshots files (.db, .crc32, .txt) into /var/lib/cassandra/data/mykeyspace/$table/
10. Restart service cassandra
11. Run nodetool repair mykeyspace -full, one node at a time

Result :

There are always missing rows, approximately the same quantity for each table but never the same ones. I tried to "mix up" a bit the procedure, like restoring keyspace before tokens, running nodetool refresh before repair, but I meet the same issue everytime.

Since I'm not far from having a "good" restore, I think that I'm missing something pretty obvious. Analyzing logs didn't really help, as they don't show any error/failure messages.

Any help would be welcomed :) I can of course give more information if needed.

edit : no one ? I updated the question with cassandra version (3.0.9), which I forgot in the first place. I tried again to restore, but no luck. I don't have any more idea really :(

I have a production server running Postgres 9.4. The database is > 10 GB. Is it possible to upgrade to Postgres 9.5 without downtime and without loosing data?

Upgrade tutorials recommend to stop Postgres while executing sudo pg_upgradecluster 9.4 main, but this may take a long time. Updating a 10 GB cluster can take several hours!

I also tried pg_dump mydb > db.sql. Dropping the database and inserting the dump again in PG 9.4 (psql -d mydb -f db.sql) took about 50 minutes.

But inserting the dump in PG 9.5 finished only after mor than 7 hours. Especially creating the index was really slow...

2016-07-18 00:13:55 CEST [60358-5] ERROR:  canceling autovacuum task
2016-07-18 00:13:55 CEST [60358-6] CONTEXT:  automatic analyze of table ...
2016-07-18 00:36:20 CEST [60366-1] ERROR:  canceling autovacuum task
2016-07-18 00:36:20 CEST [60366-2] CONTEXT:  automatic analyze of table ...
2016-07-18 04:21:40 CEST [60361-1] ERROR:  canceling autovacuum task
2016-07-18 04:21:40 CEST [60361-2] CONTEXT:  automatic analyze of table ...
2016-07-18 07:55:19 CEST [61316-1] ERROR:  canceling autovacuum task
2016-07-18 07:55:19 CEST [61316-2] CONTEXT:  automatic analyze of table ...

So neither pg_upgradecluster nor pg_dump is an acceptable solution. Even with PG 4 you would have a downtime of at least 50 minutes. Therefore: How can databases be upgraded on production servers or big master-slave-clusters without downtime and dataloss?

as embarrassing as it sounds I managed to dd a debian iso to an external hd instead of my usb pen drive.

now my 1.5 tb western digital has 1 700mb partition named debian and the rest is unallocated space.

if I understand correctly how dd works the first 700mb of data in the disc are lost since they have been overwritten.

is there any way to save the rest?

//the disk was ntfs partitioned

given two 2TB USB external disks that have to be combined to one 4TB volume and formatted with one big Filesystem (XFS), I have a small question to ask.

Does LVM provide better Data recovery, should one disk be unplugged/damaged by being able to recover the data of the still working disk or is everything lost?

I would appreciate a solution where only the data of one disk is lost and I can recover the content of the other with the usual filesystem/lvm/raid tools. Is that possible with LVM or RAID "linear"? This is for storing unimportant files that can be retrieved from backup, but I want to save time :)

Thank you in advance

But I don't believe it.

The machine is a Dell PowerEdge 2600 server running Windows Server 2008 trial 32bit (yah, its not supposed to...but it works! [well, it used to]).

For the sake of confusion: the drives are numbered 0, 1 and 2.

I was coding away as usual when I noticed the Dell logo on the front of the case was orange. So I opened the case door and saw that the HD vents were completely covered with dust (I know its not related to the orange light...but I hate dust). Since the drives are hot-swappable I yanked drive 2 out and cleaned off the dust and put it back in. I then yanked drive 1 out and cleaned the dust off that one and put it back in. Someone asked be to help set up a printer on their machine so I got up and 20 minutes later came back to see 'No boot device available - strike F1 to retry boot, F2 for setup utility` displayed on the server's monitor. I look down at the drives and drives 1 and 2 have orange lights instead of the green ones!

Since then here is what I have tried:

• Installed drives into a Dell PowerEdge 2500. Drives were detected fine. Got a message stating Missing operating system.
• Reset the bios on the original PowerEdge 2600 (pulled the bios battery out). All drives appear fine. Get the Missing operating system message when booting. A drive lights are green.
• Booted Ubuntu from a CD to inspect the drives. 2 of the drives are displayed in Computer. Since the data is striped the files/folders in the drives are gibberish.
• Booted Ubuntu and opened Terminal and executed sudo fdisk -l which listed the 3 drives. On the 3rd drive listed it states Disk identifier: 0x00000000 Disk /dev/sdb doesn't contain a valid partition table

Do you think the drives ARE actually toast?
Could it be SCSI or other hardware failure?
Could it be incorrect System Settings? Is there any way to create a virtual RAID in Ubuntu on the 2 drives that are "valid" so I can copy the data to a network share?
Should I try reinstalling the Windows Server OS(eek!)? Do you have any suggestions that I can try?

UPDATE

After doing lots of googling I came across Raid Reconstructor. I tried this program using my Dell PowerEdge 2600 using a bootable windows XP CD but it did not work (no drives detected). I then installed two of the drives into the PowerEdge 2500 alongside the 2500's existing single-drive RAID 0 running Microsoft Server 2003. I then installed and activated Raid Reconstructor which created a virtual image of the RAID-5 array, opened the image with Captain Nemo, and backed up my C:/Websites directory to another computer...with ALL files intact (so far)!!!

I will hopefully be able to restore the drives 100%

Lessons learned:

• I don't care if the server can "hot-swap" drives. DON'T FREAKING DO IT!
• Back up your data, dummy!

Thanks for all your help, answers, and comments (and for being wrong about the data loss. haha)!