Home / server / Questions / 175285
Accepted
Phil
Asked: 2010-08-28 06:29:26 +0800 CST

Efficient, Large Data Access Solutions (noob)

  • 772

I have a self-hosted web site that allows users to log in and view data stored in different files across a network. The web server is running a service that accesses the data and sends it to the users.

The setup is simple: The web server is a dedicated machine, networked with other data servers through a gigabit LAN. The web service is accessing the data through simple SMB file shares. It knows nothing of the network, and is just using simple filepaths like \\dataserver1\foo\bar.jpg. Each data server is storing terabytes worth of data.

I'm noticing that the data access is somewhat slower than I expected. Is this the best way to make the data available to users? Is there a better, more efficient solution than SMB file shares? This system is entirely Windows-based.

networking network-share storage

1 Answers

  1. Best Answer

    \Well, first - I would get rid of \\dataserver1\ - I really lOVE using DFS for organizing my shares in a hiararchy that does NOT contain the server name. Allows replication etc. at later stages.

    That said - you are basically done. YOu need to find out where your bottlenecks are. On 2008R2 file shares are not particularly slow. But for large files, the network may be slow (gigabit lan - sorry - is not exactly fast, 80mbite per second is all you will get here). You may also have memory issues or overworked disc IO - so I suggest making a performance analysis on your storage servers.

    Can you please update your question with a more detailed analysis? Given "terabyte of data" I assume you may run into a problem with your web server saturating at least it's own side of tne network, in which case getting a switch with at least 1-2 10gbit ports is the only solution (or a card that allows binding). Even if individual servers do not serve more than 1gbit - the single web server side may be overloaded.

    How much processing happens? How muc hdata is "thrown away" at the web server? If that is a lot, replace the SMB shares with web services on the servers that PRE-FILTER THE DATA, so the throw away part happens on the individual storage servers.

    eally depends on waht you DO with it, and you dont give us enough information here to maake more than totally uneducated wild guesses.

    • 1