I need to change group membership for a bunch of users.
How do I
nogroup primary group?Basically I need to change all the users that are currently in the clients group to nogroup and add clients as an auxiliary group to them.
Can this be done via shell without individually editing each user?
Our FreeNAS is running; I can login via ssh.
But the Web UI is not responding.
From How to restart web ui on freenas I tried
service nginx restart
which gave:
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Stopping nginx.
Waiting for PIDS: 2180.
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
then service django restart and got:
cat: /var/run/django.pid: No such file or directory
usage: kill [-s signal_name] pid ...
kill -l [exit_status]
kill -signal_name pid ...
kill -signal_number pid ...
Traceback (most recent call last):
File "/usr/local/www/freenasUI/manage.py", line 59, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 219, in execute
self.validate()
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/usr/local/lib/python2.7/site-packages/django/core/management/validation.py", line 35, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/usr/local/lib/python2.7/site-packages/django/db/models/loading.py", line 146, in get_app_errors
self._populate()
File "/usr/local/lib/python2.7/site-packages/django/db/models/loading.py", line 61, in _populate
self.load_app(app_name, True)
File "/usr/local/lib/python2.7/site-packages/django/db/models/loading.py", line 78, in load_app
models = import_module('.models', app_name)
File "/usr/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/usr/local/www/freenasUI/freeadmin/models.py", line 29, in <module>
from south.modelsinspector import add_introspection_rules
File "/usr/local/lib/python2.7/site-packages/South-0.7.3-py2.7.egg/south/modelsinspector.py", line 17, in <module>
from django.contrib.contenttypes import generic
File "/usr/local/lib/python2.7/site-packages/django/contrib/contenttypes/generic.py", line 13, in <module>
from django.contrib.admin.options import InlineModelAdmin, flatten_fieldsets
File "/usr/local/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 4, in <module>
from django.contrib.admin.options import ModelAdmin, HORIZONTAL, VERTICAL
File "/usr/local/lib/python2.7/site-packages/django/contrib/admin/options.py", line 8, in <module>
from django.contrib import messages
File "/usr/local/lib/python2.7/site-packages/django/contrib/messages/__init__.py", line 1, in <module>
from api import *
EOFError: EOF read where object expected
Starting django
GUI is still not working. Should I try reboot ? Or is there another way?
We are running FreeNAS 8.2.0 (specs below), and getting extremely slow FTP performance. We have a 100/70mbps FIOS connection, yet FTP (remote) from the FreeNAS is never more than about 5mbps.
We tested FTP from a Mac as a server, and got around 40mbps, but it only streams for 10 seconds or so, then drops to 0 for a minute before resuming.
We tested with iperf -c iperf.scottlinux.com -P20 from the FreeNAS and got 60mbps upstream. Same from the Mac. So it seems the network is working fine.
Tried AFP and SMB, but results were worse than FTP.
Is there anything we can do? Do we need a stronger machine?
(Local transfers over LAN go at 800mbps or more, so I know the actual storage [zfs] is fast enough.)
System Information
Build: FreeNAS-8.2.0-RELEASE-p1-x64 (r11950)
Platform: Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz
Memory: 6121MB
Load Average: 0.00, 0.01, 0.00
Our FreeNAS server log is constantly filling up with
Apr 5 15:13:49 Wheelhouse NAS kernel: swap_pager: I/O error - pagein failed; blkno 524312,size 4096, error 6
Apr 5 15:13:49 Wheelhouse NAS kernel: vm_fault: pager read error, pid 1 (init)
Apr 5 15:13:49 Wheelhouse NAS kernel: swap_pager: I/O error - pagein failed; blkno 524312,size 4096, error 6
Apr 5 15:13:49 Wheelhouse NAS kernel: vm_fault: pager read error, pid 1 (init)
Apr 5 15:13:49 Wheelhouse NAS kernel: swap_pager: I/O error - pagein failed; blkno 524312,size 4096, error 6
Apr 5 15:13:49 Wheelhouse NAS kernel: vm_fault: pager read error, pid 1 (init)
Apr 5 15:13:49 Wheelhouse NAS kernel: swap_pager: I/O error - pagein failed; blkno 524312,size 4096, error 6
and so on.
What can we do?
It's already filled up /var/log so that /var is "109%" full! Can I stop the logging somehow?
We are currently replacing a bad drive in one of the RAIDZs...
> zpool status
pool: raid-5x3
state: ONLINE
scrub: scrub completed after 15h52m with 0 errors on Sun Mar 30 13:52:46 2014
config:
NAME STATE READ WRITE CKSUM
raid-5x3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada5p2 ONLINE 0 0 0
gptid/a767b8ef-1c95-11e2-af4c-f46d049aaeca ONLINE 0 0 0
ada8p2 ONLINE 0 0 0
ada10p2 ONLINE 0 0 0
ada7p2 ONLINE 0 0 0
errors: No known data errors
pool: raid2
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h57m, 4.48% done, 20h24m to go
config:
NAME STATE READ WRITE CKSUM
raid2 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
gptid/5f3c0517-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
replacing DEGRADED 0 0 0
gptid/5fe33556-3ff2-11e2-9437-f46d049aaeca UNAVAIL 0 0 0 cannot open
ada0 ONLINE 0 0 0 113G resilvered
gptid/60570005-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/60ebeaa5-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/61925b86-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
errors: No known data errors
Upon logging into our FreeNAS via shell, it says
/var: write failed, filesystem is full
So I did
> df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/FreeNASs2a 927M 358M 494M 42% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 4.6M 1.8M 2.4M 43% /etc
/dev/md1 824K 2.5K 756K 0% /mnt
/dev/md2 149M 149M -12M 109% /var
/dev/ufs/FreeNASs4 20M 1.7M 17M 9% /data
raid-5x3 100G 58M 100G 0% /mnt/raid-5x3
raid-5x3/alpha 11T 11T 100G 99% /mnt/raid-5x3/alpha
raid2 11T 9.8T 903G 92% /mnt/raid2
The biggest folder in /var was log ... and the biggest file in there was messages (146669553 bytes) so I rm messages.
But df still shows the same exact thing, and I'm still getting the error.
I need to replace a bad disk in a zpool on FreeNAS.
zpool status shows
pool: raid-5x3
state: ONLINE
scrub: scrub completed after 15h52m with 0 errors on Sun Mar 30 13:52:46 2014
config:
NAME STATE READ WRITE CKSUM
raid-5x3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada5p2 ONLINE 0 0 0
gptid/a767b8ef-1c95-11e2-af4c-f46d049aaeca ONLINE 0 0 0
ada8p2 ONLINE 0 0 0
ada10p2 ONLINE 0 0 0
ada7p2 ONLINE 0 0 0
errors: No known data errors
pool: raid2
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
raid2 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
gptid/5f3c0517-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/5fe33556-3ff2-11e2-9437-f46d049aaeca UNAVAIL 0 0 0 cannot open
gptid/60570005-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/60ebeaa5-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/61925b86-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
errors: No known data errors
glabel status shows
Name Status Components
ufs/FreeNASs3 N/A da0s3
ufs/FreeNASs4 N/A da0s4
ufsid/4fa405ab96518680 N/A da0s1a
ufs/FreeNASs1a N/A da0s1a
ufs/FreeNASs2a N/A da0s2a
gptid/5f3c0517-3ff2-11e2-9437-f46d049aaeca N/A ada1p2
gptid/60570005-3ff2-11e2-9437-f46d049aaeca N/A ada3p2
gptid/60ebeaa5-3ff2-11e2-9437-f46d049aaeca N/A ada4p2
gptid/a767b8ef-1c95-11e2-af4c-f46d049aaeca N/A ada6p2
gptid/61925b86-3ff2-11e2-9437-f46d049aaeca N/A ada9p2
gptid/4599731b-8f15-11e1-a14c-f46d049aaeca N/A ada10p2
camcontrol devlist shows
<Hitachi HDS723030BLE640 MX6OAAB0> at scbus0 target 0 lun 0 (pass0,ada0)
<ST3000VX000-9YW166 CV13> at scbus4 target 0 lun 0 (pass1,ada1)
<ST3000VX000-9YW166 CV13> at scbus6 target 0 lun 0 (pass3,ada3)
<Hitachi HDS723030BLE640 MX6OAAB0> at scbus7 target 0 lun 0 (pass4,ada4)
<ST3000DM001-9YN166 CC4C> at scbus8 target 0 lun 0 (pass5,ada5)
<WDC WD30EZRX-00MMMB0 80.00A80> at scbus8 target 1 lun 0 (pass6,ada6)
<WDC WD30EZRX-00MMMB0 80.00A80> at scbus9 target 0 lun 0 (pass7,ada7)
<ST3000DM001-9YN166 CC4C> at scbus9 target 1 lun 0 (pass8,ada8)
<Hitachi HDS723030BLE640 MX6OAAB0> at scbus10 target 0 lun 0 (pass9,ada9)
<Hitachi HDS5C3030ALA630 MEAOA580> at scbus11 target 0 lun 0 (pass10,ada10)
< USB Flash Memory 1.00> at scbus12 target 0 lun 0 (pass11,da0)
I'm pretty sure that ada2 is the bad disk.
It appears that I left a spare in there - ada0 - last time I was in the box. Can I replace ada2 with ada0 remotely? Until someone gets to the office? With what commands?
Here's what I don't understand:
glabel status?zpool status show those long gptid's for some disks, and "ada" names for others?zpool replace raid2 -- what do I use for the device and new-device names?
I need to replace a bad disk in a zpool on FreeNAS.
zpool status shows
gptid/5fe33556-3ff2-11e2-9437-f46d049aaeca UNAVAIL 0 0 0 cannot open
How do I find the serial # of that disk?
More:
glabel status gives:
Name Status Components
ufs/FreeNASs3 N/A da0s3
ufs/FreeNASs4 N/A da0s4
ufsid/4fa405ab96518680 N/A da0s1a
ufs/FreeNASs1a N/A da0s1a
ufs/FreeNASs2a N/A da0s2a
gptid/5f3c0517-3ff2-11e2-9437-f46d049aaeca N/A ada1p2
gptid/60570005-3ff2-11e2-9437-f46d049aaeca N/A ada3p2
gptid/60ebeaa5-3ff2-11e2-9437-f46d049aaeca N/A ada4p2
gptid/a767b8ef-1c95-11e2-af4c-f46d049aaeca N/A ada6p2
gptid/61925b86-3ff2-11e2-9437-f46d049aaeca N/A ada9p2
gptid/4599731b-8f15-11e1-a14c-f46d049aaeca N/A ada10p2
I don't see gptid/5fe33556 in there.
And camcontrol devlist gives:
<Hitachi HDS723030BLE640 MX6OAAB0> at scbus0 target 0 lun 0 (pass0,ada0)
<ST3000VX000-9YW166 CV13> at scbus4 target 0 lun 0 (pass1,ada1)
<ST3000VX000-9YW166 CV13> at scbus6 target 0 lun 0 (pass3,ada3)
<Hitachi HDS723030BLE640 MX6OAAB0> at scbus7 target 0 lun 0 (pass4,ada4)
<ST3000DM001-9YN166 CC4C> at scbus8 target 0 lun 0 (pass5,ada5)
<WDC WD30EZRX-00MMMB0 80.00A80> at scbus8 target 1 lun 0 (pass6,ada6)
<WDC WD30EZRX-00MMMB0 80.00A80> at scbus9 target 0 lun 0 (pass7,ada7)
<ST3000DM001-9YN166 CC4C> at scbus9 target 1 lun 0 (pass8,ada8)
<Hitachi HDS723030BLE640 MX6OAAB0> at scbus10 target 0 lun 0 (pass9,ada9)
<Hitachi HDS5C3030ALA630 MEAOA580> at scbus11 target 0 lun 0 (pass10,ada10)
< USB Flash Memory 1.00> at scbus12 target 0 lun 0 (pass11,da0)
Does that mean that ada2 is the bad disk?
UPDATE:
OK, I'm pretty sure that ada2 is the bad disk. And I have my notes, so I know which disk that is.
But it appears that I left a spare in there - ada0 - last time I was in the box. Can I replace ada2 with ada0 remotely? Until someone gets to the office?
On our FreeNAS server, zpool status gives me:
pool: raid2
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
raid2 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
gptid/5f3c0517-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/5fe33556-3ff2-11e2-9437-f46d049aaeca ONLINE 3 1.13M 0
gptid/60570005-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/60ebeaa5-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
gptid/61925b86-3ff2-11e2-9437-f46d049aaeca ONLINE 0 0 0
errors: No known data errors
What should I do? scrub the pool?
Now that our server is set up, I would like to rename our zpools.
I read somewhere that the way to do this is export & re-import under the new name. But will all my mount points and shares remain intact? Or will I have to set those up again?
We currently have a server with 5x3TB disks configured as a RAIDZ:
zpool status
pool: raid-5x3
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
raid-5x3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
replacing ONLINE 0 0 0
ada2p2 ONLINE 0 0 0
gptid/a767b8ef-1c95-11e2-af4c-f46d049aaeca ONLINE 0 0 0
ada4p2 ONLINE 0 0 0
ada5p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0
errors: No known data errors
We want to add 5 more 3TB disks into the same machine, and enlarge the current volume ... is that possible?
I am still confused by the terminology -- vdev/zpool/volume/dataset -- want to make sure I do it right.
As I understand it, I will need to make a new RAIDZ out of the 5 new disks -- is that a vdev? Then add that vdev to the current pool, and we will end up with 10x3TB - 2x3TB = 24TB of storage, where each set of 5 disks can sustain 1 disk failure.
Just not sure what the exact steps are to do this !
Also -- not sure why zpool status still says "replacing" -- it's been like that for several weeks, since I replaced a disk with a hot spare. Is it safe to remove the old disk now?
Thanks!
I have 6 drives in a NAS box. Two of them are Seagates, and they return high RAW_VALUES for errors; see below.
My other drives show far lower values.
Is this cause for alarm? Or just the way Seagates report?
Look at Raw_Read_Error_Rate and Seek_Error_Rate:
# smartctl -a /dev/ada1
=== START OF INFORMATION SECTION ===
Device Model: ST3000DM001-9YN166
Serial Number: W1F09S26
LU WWN Device Id: 5 000c50 0456076fc
Firmware Version: CC4C
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Sat Aug 18 17:34:24 2012 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 111 099 006 Pre-fail Always - 34053632
3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 32
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 060 055 030 Pre-fail Always - 21480133713
9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 2696
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 32
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 064 061 045 Old_age Always - 36 (Min/Max 34/38)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 28
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 63
194 Temperature_Celsius 0x0022 036 040 000 Old_age Always - 36 (0 19 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 43748536879750
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2867098636991
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 17478042509157
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Self-test routine in progress 90% 2696 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
What is a "hot spare" with regard to ZFS RAIDZ on FreeNAS?
I'm hoping that it means an extra, attached disk that can automatically take over for a failed disk in the RAID ... ?
If so, how do I set that up?
Sorry for the long narrative, but I'm thoroughly confused.
I'm using FreeNAS-8.0.4-RELEASE-p2-x64 (11367) on a box with 5x3TB SATA disks configured as a raidz volume.
A few days ago, the console gave me this alert:
CRITICAL: The volume raid-5x3 (ZFS) status is DEGRADED
zpool status gave:
pool: raid-5x3
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: resilver completed after 3h25m with 7607009 errors on Sun Aug 12 06:26:44 2012
config:
NAME STATE READ WRITE CKSUM
raid-5x3 DEGRADED 0 0 7.29M
raidz1 DEGRADED 0 0 14.7M
ada0p2 ONLINE 0 0 0
10739480653363274060 FAULTED 0 0 0 was /dev/ada1p2
ada2p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 3 254M resilvered
ada1p2 ONLINE 0 0 0
errors: 7607009 data errors, use '-v' for a list
I did a zpool status -v and got:
Permanent errors have been detected in the following files:
and it listed 2,660 files (out of 50,000 or so)
plus things like:
raid-5x3/alpha:<0x0>
raid-5x3/alpha:<0xf5ec>
raid-5x3/alpha:<0xf5ea>
We turned the server off, put in a new drive, in addition to the five already in there.
Went to console and view disks, it just said "loading" forever *couldn't get to the "Replace" option!
Then we got:
zpool status -v
pool: raid-5x3
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
raid-5x3 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ada0p2 ONLINE 0 0 0
10739480653363274060 UNAVAIL 0 0 0 was /dev/ada1p2
ada2p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
raid-5x3/alpha:<0x0>
/mnt/raid-5x3/alpha/staff/Sound FX jw/Sound FX - scary horror/11 DR-EerieAct3-Waterphone..aif
/mnt/raid-5x3/alpha/staff/Wheelhouse Shoots/ROCKY_THE_MUSICAL/ SHOOTS/WESTPORT/Cannon-CARD-B/CONTENTS/CLIPS001/AA0876/AA087601.SIF
... then 2,860 files and "raid-5x3/alpha:<....>" entries ...
camcontrol devlist:
<ST3000DM001-9YN166 CC4C> at scbus4 target 0 lun 0 (ada0,pass0)
<WDC WD30EZRX-00MMMB0 80.00A80> at scbus4 target 1 lun 0 (aprobe1,pass6,ada4)
<WDC WD30EZRX-00MMMB0 80.00A80> at scbus5 target 0 lun 0 (ada1,pass1)
<ST3000DM001-9YN166 CC4C> at scbus5 target 1 lun 0 (ada2,pass2)
<ASUS DRW-24B1ST a 1.04> at scbus6 target 0 lun 0 (cd0,pass3)
<Hitachi HDS5C3030ALA630 MEAOA580> at scbus7 target 0 lun 0 (ada3,pass4)
< USB Flash Memory 1.00> at scbus8 target 0 lun 0 (da0,pass5)
gpart show
=> 63 7831467 da0 MBR (3.7G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 3926538 - free - (1.9G)
=> 0 1930257 da0s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)
=> 34 5860533101 ada0 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada1 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada2 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada3 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada4 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
glabel status
Name Status Components
ufs/FreeNASs3 N/A da0s3
ufs/FreeNASs4 N/A da0s4
ufs/FreeNASs1a N/A da0s1a
gptid/446dd91d-8f15-11e1-a14c-f46d049aaeca N/A ada4p1
gptid/447999cb-8f15-11e1-a14c-f46d049aaeca N/A ada4p2
Seemed the new drive wasn't connected properly?
Re-attached it and rebooted.
Now console showed green light alert.
But when I went to "View All Volumes", it just said "Loading..."
Then:
glabel status
Name Status Components
ufs/FreeNASs3 N/A da0s3
ufs/FreeNASs4 N/A da0s4
ufs/FreeNASs1a N/A da0s1a
camcontrol devlist: Code: at scbus0 target 0 lun 0 (ada0,pass0) at scbus4 target 0 lun 0 (ada1,pass1) at scbus4 target 1 lun 0 (ada2,pass2) at scbus5 target 0 lun 0 (ada3,pass3) at scbus5 target 1 lun 0 (ada4,pass4) at scbus6 target 0 lun 0 (cd0,pass5) at scbus7 target 0 lun 0 (ada5,pass6) < USB Flash Memory 1.00> at scbus8 target 0 lun 0 (da0,pass7)
gpart show
=> 63 7831467 da0 MBR (3.7G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 3926538 - free - (1.9G)
=> 0 1930257 da0s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)
=> 34 5860533101 ada1 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada2 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada3 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada4 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
=> 34 5860533101 ada5 GPT (2.7T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338703 2 freebsd-zfs (2.7T)
zpool status:
pool: raid-5x3
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
raid-5x3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
ada2p2 ONLINE 0 0 2
ada4p2 ONLINE 0 0 0
ada5p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0
errors: 7607009 data errors, use '-v' for a list
At this point, someone on the FreeNAS forums said, "You're screwed, at some point you had 2 disks fail, bye bye data."
Is this true?
I clicked the 'scrub' button ... zpool status showed "resilver in progress .... 900h to go" ... which is like a month... and which kept going up to 30,000hrs...
Cut to: today, we rechecked all the connections on all the drives.
Then it started resilvering again, but much faster.
Several of the files – which were previously reported as corrupt – I randomly checked, and they now "seem" to be OK. (Meaning I was able to copy them and play them – most of our data is video files.)
What I'd like to do is COPY everything for which we do not have a backup, and which is not corrupt, to another machine, and then upgrade this one to RAIDZ2.
I'm thinking maybe what happened is that 2 drives became dislodged. I think the hotswap bay we have is poor quality.
But, then again, they DID appear connected, just faulted ... I don't know.
The resilver completed, in 3.5 hours.
Now zpool status says:
pool: raid-5x3
state: ONLINE
scrub: resilver completed after 3h31m with 0 errors on Fri Aug 17 21:46:12 2012
config:
NAME STATE READ WRITE CKSUM
raid-5x3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
ada2p2 ONLINE 0 0 0 236G resilvered
ada4p2 ONLINE 0 0 0
ada5p2 ONLINE 0 0 0 252G resilvered
ada3p2 ONLINE 0 0 0
errors: No known data errors
Does this mean the data is recovered?? "No known errors" sounds promising!
I've now initiated a scrub. (8 hours to go.)
We don't have a backup for ALL the data ... so we need to figure out which of those files are corrupt, and which are usable.
Did a drive fail? If so, which one? Or did it just come loose?
Do I need to replace one? Two?
Is any of our data safe? If so, which files?
My NAS box is set to default scrub threshold every 35 days.
How do I change it to scrub every Saturday night? Do I need to create a cron job?
I'm on FreeNAS-8.0.4-RELEASE-p2-x64 (11367)