Tim's questions

https://en.wikipedia.org/wiki/Clustered_file_system#Network-attached_storage says

Network-attached storage (NAS) provides both storage and a file system, like a shared disk file system on top of a storage area network (SAN). NAS typically uses file-based protocols (as opposed to block-based protocols a SAN would use) such as NFS (popular on UNIX systems), SMB/CIFS (Server Message Block/Common Internet File System) (used with MS Windows systems), AFP (used with Apple Macintosh computers), or NCP (used with OES and Novell NetWare).

https://en.wikipedia.org/wiki/Network-attached_storage

Network-attached storage (NAS) is a file-level (as opposed to block-level) computer data storage server connected to a computer network providing data access to a heterogeneous group of clients. NAS is specialized for serving files either by its hardware, software, or configuration. It is often manufactured as a computer appliance – a purpose-built specialized computer.[nb 1] NAS systems are networked appliances that contain one or more storage drives, often arranged into logical, redundant storage containers or RAID. Network-attached storage removes the responsibility of file serving from other servers on the network. They typically provide access to files using network file sharing protocols such as NFS, SMB, or AFP.

... A clustered NAS is a NAS that is using a distributed file system running simultaneously on multiple servers. The key difference between a clustered and traditional NAS is the ability to distribute[citation needed] (e.g. stripe) data and metadata across the cluster nodes or storage devices. Clustered NAS, like a traditional one, still provides unified access to the files from any of the cluster nodes, unrelated to the actual location of the data.

Does a NAS provide both block level and file level operations?

Does "Network-attached storage (NAS) provides both storage and a file system, like a shared disk file system on top of a storage area network (SAN)" mean that NAS and a shared disk file system on a SAN are the same?

Does "NAS typically uses file-based protocols (as opposed to block-based protocols a SAN would use) such as NFS (popular on UNIX systems), SMB/CIFS (Server Message Block/Common Internet File System) (used with MS Windows systems), AFP (used with Apple Macintosh computers), or NCP (used with OES and Novell NetWare)" mean that NAS and a distributed filesystem are the same?

Thanks.

Does the concept of a distributed filesystem require a distributed filesystem to consist of multiple filesystems located on different computers?

Does an NFS filesystem consist of only one filesystem? (See the diagram below)

If yes, is an NFS filesystem a distributed filesystem?

Thanks.

From Proxy server on Wikipedia:

A proxy server may reside on the user's local computer, or at any point between the user's computer and destination servers on the Internet. A proxy server that passes unmodified requests and responses is usually called a gateway or sometimes a tunneling proxy. A forward proxy is an Internet-facing proxy used to retrieve data from a wide range of sources (in most cases anywhere on the Internet). A reverse proxy is usually an internal-facing proxy used as a front-end to control and protect access to a server on a private network. A reverse proxy commonly also performs tasks such as load-balancing, authentication, decryption and caching.

From HTTP: The Definitive Guide by David Gourley and Brian Totty:

Strictly speaking, proxies connect two or more applications that speak the same protocol, while gateways hook up two or more parties that speak different protocols. A gateway acts as a “protocol converter,” allowing a client to complete a transaction with a server, even when the client and server speak different protocols.

Figure 6-2 illustrates the difference between proxies and gateways:

• The intermediary device in Figure 6-2a is an HTTP proxy, because the proxy speaks HTTP to both the client and server.

• The intermediary device in Figure 6-2b is an HTTP/POP gateway, because it ties an HTTP frontend to a POP email backend. The gateway converts web transactions into the appropriate POP transactions, to allow the user to read email through HTTP. Web-based email programs such as Yahoo! Mail and MSN Hotmail are HTTP email gateways.

In practice, the difference between proxies and gateways is blurry. Because browsers and servers implement different versions of HTTP, proxies often do some amount of protocol conversion. And commercial proxy servers implement gateway functionality to support SSL security protocols, SOCKS firewalls, FTP access, and web-based applications.

A gateway connects two parties that speak the same protocols without modification by the first source, and it connects two parties that speak different protocols and convert between the protocols by the second source. Aren’t the definitions of gateway in the two sources contradicting each other?

Nginx can be used as a reverse proxy server, when the proxied server is a CGI server, according to its official online document. A reverse proxy server is a proxy server by the first source, which connects two parties that speak the same protocols by the second source. Nginx speaks to the client using HTTP, so shouldn’t Nginx also speak to the CGI server using HTTP?

When setting up a local Wi-Fi network in a small place from an ISP, such as Verizon, using a router and modem, the machines in the network are assigned private IP addresses.

In such a local Wi-Fi network, I can use ssh name@<hostname> to access an SSH server on another computer <hostname>, where <hostname> is the output of command hostname. Is the output of hostname resolved to a private IP address, by some DNS server (possibly on the router?)?

But I heard that the output of hostname is unrelated to host name resolution by DNS. Then why can I successfully run ssh name@<hostname>, where <hostname> is the output of command hostname?

https://en.wikipedia.org/wiki/Virtual_hosting says

Name-based virtual hosts use multiple host names for the same IP address.

Does a "host name" correspond to an IP address or a pair of IP address and a port?

When specifying a "host name", do we still need to specify a port?

If that matters, consider only in virtual hosting.

Thanks.

I use rsync -a to do a full backup on a file system. Its output to stderr is empty. Shall I do some additional checking to verify if the backup is successful?

For example, how can I do a checksum to verify that?

When I do an incremental backup using rsync next time, will the way of checking by the checksum be the same? The incremental backup will have hard links to the unchanged files in the full backup, and will that make checksums of the source and the incremental backup different although the incremental backup is successful?

My OS is Ubuntu 12.04. Thanks.

I was wondering what are the best ways on local Windows for resuming transfer of files over ssh from remote linux?

I tried SSHSecureShellClient-3.2.9, but I cannot find it works for resuming.

I tried Putty's psftp. It also seems not having the resuming ability.

************** update ****************

When using WinSCP to transfer many files, if I cancel the transfer in the halfway, will the files not transferred yet and not fully transferred be saved so that I can resume their transfer next time when I get a faster connection? I am worried if I have to manually compare the source and destination files to figure out what files are already transferred and what are not.

I just learned that cpio has three modes: copy-out, copy-in and pass-through.

I was wondering what are the advantages and disadvantages of cpio under copy-out and copy-in modes over tar. When is it better to use cpio and when to use tar?

Similar question for cpio under pass-through mode versus cp.

Thanks and regards!

I am a newbie to both network and Linux. I am now trying to connect to a WEP wireless network by command line on my Ubuntu 8.10, because the Network Manager does not support 64 bit WEP.

(1) I firstly bring down the Network Manager and then try to connect to a wireless network, whose essid is candy and password is 5673212741. But it fails as shown in the following. The signal is not very strong, but I can connect to it in Windows 7. I wonder why and how to do it correctly for Ubuntu?

$ sudo /etc/init.d/NetworkManager stop  
 * Stopping network connection manager NetworkManager        [ OK ]   
$ sudo iwconfig wlan0 essid candy  
 opendo iwconfig wlan0 key 18018ce78e open  
$ sudo iwconfig wlan0 key 5673212741 open   
$ sudo  dhclient wlan0  
There is already a pid file /var/run/dhclient.pid with pid 9971  
killed old client process, removed PID file  
Internet Systems Consortium DHCP Client V3.1.1  
Copyright 2004-2008 Internet Systems Consortium.  
All rights reserved.  
For info, please visit http://www.isc.org/sw/dhcp/  

wmaster0: unknown hardware address type 801  
wmaster0: unknown hardware address type 801  
Listening on LPF/wlan0/00:0e:9b:cd:4e:18  
Sending on   LPF/wlan0/00:0e:9b:cd:4e:18  
Sending on   Socket/fallback  
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 7  
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 12  
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 20  
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 13  
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 9  
No DHCPOFFERS received.  
No working leases in persistent database - sleeping.  
$ ping www.bbc.co.uk  
ping: unknown host www.bbc.co.uk

(2) A less important question: why the scan for wireless networ does not work after I bring down the Network Manager?

$ sudo /etc/init.d/NetworkManager stop  
 * Stopping network connection manager NetworkManager         [ OK ]   
$ sudo iwlist wlan0 scan  
wlan0     Interface doesn't support scanning : Network is down      

Thanks and regards!

I am running a bash script on a Linux server. At the time it finished running, I got this message

../scripts/makepbs.sh: error reading input file: Stale NFS file handle

What does it mean?

Thanks and regards!

I have asked a question before in here. My question was that I tried to give my password to scp via piping

echo mypassword | scp [email protected]:project/* ~/project/  

However it still asks me to manually input the password. How should I specify mypassword to scp in commands?

I still don't understand one of the replies. what is a input stream by TTY, for example? What are the common ways for a input stream? How to know what type of input stream a stdin input to a command is? For example that of ssh/scp.

I have some questions regarding generating keys for ssh access:

(1) Supposed there are two computers running ssh server service and I have generated a pair of key files on computer A and copy the public file to computer B. Is it true that this is only a one-way key: We only gave computer A permission to access computer B, not gave computer B permission to access computer A? If I now want to ssh from computer B to computer A, must I generat another pair of key files on computer B and copy the public file to computer A?

(2) If I would like to connect a single local computer to several remote servers, is it to generate a common pair of key files only once on the local and copy the same public file to the remote servers, or to generate different pair of key files on the local for different remote servers?

(3) If I would like to connect several local computers to a single remote server, when copying the public files from different local computers to the remote server, is it to combine them together into a single authorized_keys file or store them in different authorized_keys files?

(4) If there are several servers shared the same file system by, for example, NFS, how to generate keys and arrange the key files for accessing from one server to the other? Also how to still generate keys and arrange the key files for a local computer to access anyone of the servers?

All the machines above are Linux.Please provide examples and commands in your reply so that I can better understand how to solve the problems.

Thanks and regards!

I have an attached screen session where I am running some jobs. I closed its terminal without detaching it because it does not respond suddently. Now by screen -ls I can see it is still attached. But I cannot reattach it without detaching it first, however detaching it takes a very long time and still is not finished yet.

  bash-3.2$ screen -ls  
  There are screens on:  
   14493.ssh2others (Attached)  
  1 Socket in /tmp/uscreens/S-tim.  

  bash-3.2$ screen -r 14493  
  There is a screen on:  
   14493.ssh2others (Attached)  
  There is no screen to be resumed matching 14493.  
  bash-3.2$ screen -d 14493  

I wonder why it takes so long and is not finished detaching yet?

Since my jobs are running in the screen session, I am afraid killing it will kill my jobs as well. So what can I do to detach and reattach it now?

Thanks!

I have some process that is in uninterruptible sleep due to limited memory. After I kill other processes, there is enough memory available for it now. How do I change the status of the process from uninterruptible sleep to normal running? Or the OS will automatically make the change? Or do I have to kill it and start it agian? Thanks and regards!

I am running a bash script as a background job. The bash script calls a time-consuming executable. If I am not wrong, the running of the bash script is the parent process and the running of the executable is the child process.

(1) when I look at the output of top, it only shows the child process which is the running of the executable, not shows the parent process which is the running of the script. Just wonder why not show both? And how to show both?

(2) I now want to stop the whole running by killing the parent process which is the background job

kill -9 $(jobs -p)

The terminal shows that the running of the bash script is killed. But the running of the executable still hangs on the output of top. I just wonder how to kill both parent and child process?

Thanks and regards!

I failed in ssh connection to a server. Here is the output of connection

$ ssh -v [server]
OpenSSH_5.1p1 Debian-3ubuntu1, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /home/tim/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to [server] [server ip] port 22.
debug1: Connection established.
debug1: identity file /home/tim/.ssh/identity type -1
debug1: identity file /home/tim/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/tim/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host`

Anyone know what's wrong? Thanks and regards!

I had been running a time-consuming background program via a bash script on a linux server. In the same bash script, I set the notification by "set -o notify" so I could know when the background job was done.

Probably yesterday night the notification popped out to say the background job was done, and after that there were following error messages, which I am not sure if occured immediately after the background job finished or a while later, since I was not at my terminal at that time:

do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused

do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused

do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused

I also redirected the stdout output of my background job to a log file which seems to say the program was not finished as expected but terminated midway.

Could you explain the meaning of the error message? If possible,what kinds of problem could I possibly met? Could it be that the administrator placed some restriction on the resources that I can use on that server?

Thanks and regards!

UPDATE:

the same hard drive is mounted via NFS over several servers including the one mentioned above. I just found a similar but different error occur on the other server:

do_ypcall: clnt_call: RPC: Timed out

This error and the previous error seem not affect the programs that are running. BTW my program has IO operations.

Looking at the output of top on our server, one of my colleagues told me that the fact that some processes got less than 100 "%CPU" was because I was running too many processes. He added that based on his experience if I run less than 6 processes, then probably all the processes would have 100 "%CPU".

I don't want to be an annoyance to other users, but I doubt what he said is correct. The server has 16 cores and the current load average is between 10 and 11. From what I have learned, it is not overloaded. But I don't know why some processes are just getting less than 100 "%CPU"? Is it really because of me?

Thanks and regards!

Here comes the output of top:

top - 16:34:13 up 32 days,  1:36, 12 users,  load average: 10.61, 10.39, 10.22
Tasks: 380 total,  10 running, 370 sleeping,   0 stopped,   0 zombie
Cpu(s): 55.0%us,  1.7%sy,  0.0%ni, 42.2%id,  0.5%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:  130766620k total, 39859784k used, 90906836k free,   849412k buffers
Swap: 47351548k total,   279456k used, 47072092k free, 19792956k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                        
17197 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4510:11 MLtest                                                                                       
28762 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4633:01 MLtest                                                                                       
29249 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4623:03 MLtest                                                                                       
29560 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4626:59 MLtest                                                                                       
 4904 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4757:12 MLtest                                                                                       
 5143 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4759:40 MLtest                                                                                       
29389 tim    18  -2 1315m 1.3g 1504 R   99  1.0   4622:11 MLtest                                                                                       
 5285 tim    18  -2 1315m 1.3g 1504 R   97  1.0   4758:49 MLtest                                                                                       
 4763 tim    18  -2 1315m 1.3g 1504 R   93  1.0   4754:22 MLtest                                                                                       
 9456 zma    18  -2  206m  85m  11m S   48  0.1  60:46.78 dropbox                                                                                         
 7527 vals   18  -2 1266m 436m  42m S    4  0.3 613:57.10 MATLAB                                                                                          
 2903 root   15  -5     0    0    0 S    1  0.0  19:00.01 rpciod/0                                                                                        
19133 vals   18  -2 1380m 503m  42m S    1  0.4 798:47.99 MATLAB                                                                                          
12454 tim    18  -2 19248 1588 1024 R    1  0.0   0:48.88 top                                                                                             
   12 root   RT  -5     0    0    0 S    1  0.0  35:01.05 migration/3                                                                                     
 2924 root   15  -5     0    0    0 S    1  0.0  27:20.92 nfsiod                                                                                          
12690 jun    18  -2  913m  84m 2684 S    1  0.1 121:55.65 MATLAB                                                                                          
19650 jun    18  -2 19244 1600 1028 S    1  0.0   6:58.41 top                                                                                             
6 root       RT  -5     0    0    0 S    0  0.0 129:49.45 migration/1                                                                                     
9 root       RT  -5     0    0    0 S    0  0.0 104:34.66 migration/2                                                                                     
 2870 daemon 20   0  8180  404  308 S    0  0.0   5:18.91 portmap                                                                                         
 8985 root   20   0 28484  344  264 S    0  0.0   6:24.77 hald-addon-stor                                                                                 
 9293 root   20   0  369m 4208 2316 S    0  0.0  83:36.35 kdm_greet                                                                                       
24028 tim    18  -2  871m 140m  45m S    0  0.1   7:50.56 MATLAB                                                                                          
1 root      20   0  4104  304  224 S    0  0.0   0:03.59 init
2 root      15  -5     0    0    0 S    0  0.0   0:00.26 kthreadd
3 root      RT  -5     0    0    0 S    0  0.0   0:00.31 migration/0
4 root      15  -5     0    0    0 S    0  0.0   1:08.91 ksoftirqd/0

I am using a linux server which has 128GB of memory and 24 cores. I use top to see how much it is used. Its output is pasted at the end of the post. Here are two questions:

(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to achieve almost 100%, doesn't it?

(2) how to understand the load average on the first line ("load average: 14.04, 14.02, 14.00")?

Thanks and regards!

Edit:

Thanks!

I also really like to hear some rough numbers based on used percentage of memory to determine if a server is heavily loaded, since I once became the one who cramed the server without understanding the current load.

Is swap regarded as almost the same as memory? For example, when memory and swap are almost of same size, if the memory is almost running out but the swap is still largely free, may I just view it as if the used percentage of memory + swap is still not high and run other new processes?

How would you consider together CPU or memory (or memory + swap) usage? Do you become worried if either of them reaches too high or both?

Output of top：

$ top

 
top - 12:45:33 up 19 days, 23:11, 18 users,  load average: 14.04, 14.02, 14.00
Tasks: 484 total,  12 running, 472 sleeping,   0 stopped,   0 zombie
Cpu(s): 36.7%us, 19.7%sy,  0.0%ni, 43.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  130766620k total, 130161072k used,   605548k free,   919300k buffers
Swap: 63111312k total,   500556k used, 62610756k free, 124437752k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6529 sanchez   18  -2 1075m 219m  13m S  100  0.2  13760:23 MATLAB
13210 timothy   18  -2 48336  37m 1216 R  100  0.0   3:56.75 absurdity
13888 timothy   18  -2 48336  37m 1204 R  100  0.0   2:04.89 absurdity
14542 timothy   18  -2 48336  37m 1196 R  100  0.0   1:08.34 absurdity
14544 timothy   18  -2  2888 2076  400 R  100  0.0   1:06.14 gatherData
 6183 sanchez   18  -2 1133m 195m  13m S  100  0.2  13676:04 MATLAB
 6795 sanchez   18  -2 1079m 210m  13m S  100  0.2  13734:26 MATLAB
10178 timothy   18  -2 48336  37m 1204 R  100  0.0  11:33.93 absurdity 
12438 timothy   18  -2 48336  37m 1216 R  100  0.0   5:38.17 absurdity
13661 timothy   18  -2 48336  37m 1216 R  100  0.0   2:44.13 absurdity
14098 timothy   18  -2 48336  37m 1204 R  100  0.0   1:58.31 absurdity
14335 timothy   18  -2 48336  37m 1196 R  100  0.0   1:08.93 absurdity
14765 timothy   18  -2 48336  37m 1196 R   99  0.0   0:32.57 absurdity
13445 timothy   18  -2 48336  37m 1216 R   99  0.0   3:01.37 absurdity
28990 root      20   0     0    0    0 S    2  0.0  65:50.21 pdflush
12141 tim       18  -2 19380 1660 1024 R    1  0.0   0:04.04 top
 1240 root      15  -5     0    0    0 S    0  0.0  16:07.11 kjournald
 9019 root      20   0  296m 4460 2616 S    0  0.0  82:19.51 kdm_greet
    1 root      20   0  4028  728  592 S    0  0.0   0:03.11 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:01.01 migration/0
    4 root      15  -5     0    0    0 S    0  0.0   0:08.13 ksoftirqd/0
    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0
    6 root      RT  -5     0    0    0 S    0  0.0  17:27.31 migration/1
    7 root      15  -5     0    0    0 S    0  0.0   0:01.21 ksoftirqd/1
    8 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/1
    9 root      RT  -5     0    0    0 S    0  0.0  10:02.56 migration/2
   10 root      15  -5     0    0    0 S    0  0.0   0:00.34 ksoftirqd/2
   11 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/2
   12 root      RT  -5     0    0    0 S    0  0.0   4:29.53 migration/3
   13 root      15  -5     0    0    0 S    0  0.0   0:00.34 ksoftirqd/3

I set up a SSH server on my Ubuntu on my laptop by OpenSSH. While ssh localhost is okay, I try to first ssh to another computer from, and then from there ssh back to my laptop with its external IP address but it fails with "connect to host xxx port xxx: Connection timed out". My laptop is wirelessly connected to the internet through a router with Comcast as the provider. So I guess I am having a dynamic IP? What is the correct way for setup or connection? Thanks and regards!