Home / server / Questions / 642722
Accepted
Ztyx
Asked: 2014-11-08 04:39:23 +0800 CST

Issue with snapshot creation

  • 772

I am running into an issue when I try to repair a keyspace:

myuser@host-1:~$ nodetool repair -pr mykeyspace
[2014-11-07 13:33:10,132] Starting repair command #13, repairing 1 ranges for keyspace mykeyspace
[2014-11-07 13:33:20,668] Repair session 3baeeb40-667a-11e4-a6e9-3562b4c30e9e for range (-3651079129967449,-2] failed with error java.io.IOException: Failed during snapshot creation.
[2014-11-07 13:33:20,668] Repair command #13 finished

When I look in system.log on host-1 it says:

ERROR [RepairJobTask:1] 2014-11-07 13:33:20,665 RepairJob.java (line 125) Error occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /X.X.X.33
        at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:81)
        at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:344)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [AntiEntropySessions:12] 2014-11-07 13:33:20,666 RepairSession.java (line 288) [repair #3baeeb40-667a-11e4-a6e9-3562b4c30e9e] session completed with the following error
java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [AntiEntropySessions:12] 2014-11-07 13:33:20,667 CassandraDaemon.java (line 199) Exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        ... 3 more
ERROR [Thread-1607] 2014-11-07 13:33:20,668 StorageService.java (line 2599) Repair session 3baeeb40-667a-11e4-a6e9-3562b4c30e9e for range (-3651079129967449,-2] failed with error java.io.IOException: Failed during snapshot creation.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2591)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        ... 1 more
Caused by: java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        ... 3 more

When I look in the log on X.X.X.33 I don't see any log errors whatsoever.

Anyone who could give me pointer in the right direction?

datastax-enterprise

1 Answers

  1. Best Answer

    Fixed. I issued a manual nodetool snapshot mykeyspace on X.X.X.33 and managed to get a better error in the log. Turns out I had a missing datafile, probably from dropping a table and creating it faster than that schema change had propagated throughout the cluster. Restarted the Cassandra node (for it to discover the file was missing) and TRUNCATEd the table.

    • 2