user76976's questions

Suppose that one has 100 machines (each with a 2TB hard drive), all connected in a network.

Is there a way (in Linux) to unite this combined 200TB of disk space into one folder, that can be shared using NFS among all machines that are in the network?

Going this route, all machines will be able to read/write to the same folder, and this way, I/O will be spread evenly between them. Is this a good idea if one needs a large file system to store 100s of TB of data? (note: data will be split to many smaller files, of size ~ 500GB each).

Is there a ready-made solution (prefereable, an open source one) that achieves just that?

I have a network of 100 machines.

Is there a limit to the number of machines that can connect to one single machine (at the same time)?

For example, can I have 99 of my machines maintain continuous ssh connection to the 100th machine? Can I have every one of my machines (every one of the 100) maintain a continuous ssh connection to all other 99 machines?

How much memory does each such a connection take?

I have a network of 20 machines, running Ubuntu 10.04.

Each machine has about 200[GB] of data that I'd like to share with all other 19 machines for READ ONLY PURPOSES. The reading should be done at the fastest possible way.

A friend told me to look into setting up HTTP / FTP. Is it indeed the optimal way to share data between the machines (better than NFS)? if so, how do I go about it?

Is there a python module that would help in accessing/reading the data?

UPDATE: Just to clarify, all I want is to be able (from within machine X) to access one of machine Ys files and LOAD IT INTO MEMORY. all of the files are of uniform size (500 [KB]). Which method is fastest (SAMBA / NFS / HTTP / FTP)?

I need to build a system to store & maintain a huge amount (20 [TB]) of time-series data (for many different instruments), so that it would support the following requirements:

(1) fast appends of new data, as new data comes in
(2) fast retrievals of existing (already stored) data

There are 10,000 instruments and 1000 data fields (updated every 1-min) to save for each instrument. Once data is written to disk, it remains unchanged (no issues of concurrent writing/reading).

Since there would be no need for any joins whatsoever (typical query is: give me all instruments for field 'X' on interval 'Y'), I tend to store the data using flat binary files that will be named like this: fieldName.timeStamp.bin; this way, I would be able to store all the data in flat binary files (no need for a huge expense for a giant server / commercial database) and still, queries will be fast.

Since it's a lot of data (circa 20[TB]), I thought I would need some logic to distribute the files (fieldName.timeStamp.bin) between all of my machines. Here's what I had in mind: there will be a central machine to which all queries will be sent. this central machine (based on the field & timestamp requested) would route the query to the machine of interest, that would in turn return the requested data.

My questions are:

(1) is this design scalable as I think it is? any drawbacks?

(2) is there anything I am missing here that might hurt performance?

(3) is it really the best way to send all queries to a central machine, that would in turn route the query to the right machine? or would it be best to directly access the right machine (suppose I know which one it is ) using NFS?

(4) is there a faster way than NFS to access the right machine to read data from it? are there other methods for sharing all the data that on the data machines with client machines?

All of my machines use Ubuntu Linux. As can be understood, there will be many client machines that would access the various data machines and read (only read, not write) data from them. my goal is to have the data read as fast as possible.

I have a bunch of different binary files (total size of all files is circa ~500GB) stored in a hard drive that is mounted to machine001.

I have 200 other machines that need to access this data for READ-ONLY PURPOSES.

What is the best way to share this data with all other computers, so that READS are fastest? (I use Ubuntu Linux)

Is there a different scheme that would be even better for fastest reads?

I have about a dozen computers connected in a network. All have Ubuntu 10.04 installed.

Every computer is running unique processes, and sometimes I have the need to send a message from one computer to another. I have a python script that allows sending an email through gmail, and it works.

yet, since all computers are connected in a network, I was hoping there could be a way to send 'private' emails (from one computer to another) without the need to communicate through the internet.

is it at all possible?