Daniel Magliola's questions

I need to set up an SFTP server that, essentially, has very large capacity. I need to give one of our partners SFTP login details to a server where they will upload millions of files, totalling a few hundred Terabytes. I will then be selective and quite rarely reading some of these files. This is the only actual requirement, any technology choice is up for grabs.

What comes to mind as the easiest way is to have some sort of EC2 instance running the SFTP server in such a way that anything uploaded is either directly sent to S3, or some sort of process discovers new files when they get uploaded, copies them to S3, and deletes them from disk.

Is this the best way? Is there any other way of getting a server that essentially has "infinite and magically growing disk space"?

Thanks for your help! Daniel

I have a mobile app that uploads files to a server and needs to be able to download them. I'm using S3 to hold the files, and have a RoR backend to manage all this.

Is it possible to have the following setup?

• RoR backend uses a certain "access key and secret access key" to access the S3 API. This access key has access to listing objects in the bucket, reading them, and creating new ones.
• When the phone app uploads a file, the RoR backend saves it to S3, and generates a URL
• The phone, to download the file, accesses that URL. It has permissions to read those files, but not to write, and absolutely no permission to list (this is key)
• I'd like for the phone to just hit a URL using a normal HTTP GET and get the file, without having to deal with the S3 api. This is very desirable.
• Ideally, if I could have it so that the phone needs to also have an access key / token of some sort, but it can simply add it to the URL as a parameter, rather than go through the S3 API, that'd be great. This gives me some extra security without having to actually tie the client so much to S3, in case we want to change providers in the future.

AWS has a multitude of options for configuring these things, and it's not clear to me which is the best, and which scenarios are possible (I see there are options to generate URLs that are time-limited. This would be great in my case, if the server can give the client a URL that is only valid for a few minutes, in case the URL gets "leaked" somehow, but I haven't been able to figure out exactly how that works)

Any pointers would be greatly appreciated!

I'm testing whether some changes I made to some code are effective or not. For this, I need to know how the file system cache normally works. I'm assuming the default is what we're getting, since we probably haven't tweaked anything interesting regarding this.

Essentially, if from my code I ask "does this file exist?", the answer comes back in "x" milliseconds. If I ask again a few minutes later, the answer comes back much, much quicker, so I'm guessing there's some caching going on there.

How long should I wait if I want to test again, with the cache flushed, so it'll take "long" again? (I want to see if I fixed a performance problem in my web app, but I can only know that it's fixed if the info about files existence isn't cached, because if it is, it does run fast)

I know this is not the most eloquent question, and the answer is probably "it depends", I'm asking for a sane default. In other words, if I test again in 3 hours, or 6, or whatever, can I count on it not being cached?

As far as I can tell... This is a Ubuntu server, kernel version: 3.9.3-x86-linode52, Ubuntu version: Ubuntu 12.04.3 LTS, hosted on Linode. The file system is ext3

Thank you!
Daniel

I have a website i'm hosting in Apache (on Windows), and i'd like to protect it so that anyone going in needs to enter a password. This is not meant to be a fortress of security, I know this sucks, but it's better than nothing, and I don't want to go through any of the alternatives that'll make it more secure (they're truly not worth it for this).

What i'd like is for the typical HTTP authentication dialog to pop up asking the visitor for a password. Everyone will have the same password. In fact, if I can have the username be blank, that'd be ideal. If not, everyone will have the same username.

Could you let me know how I can do this? All tutorials I've found were for more advanced use cases, where the actual users/passwords of the machine were used.
Is it possible to just set this to a hardcoded password in a file? Or do I need to create a Windows user?

I have a site with a decently big database, 3Gb in size, a couple of tables with a dozen million records.

It's currently 100% on MyISAM, and I have the feeling that the server is going slower than it should because of too much locking, so I'd like to try going to InnoDB and see if that makes things better. However, I need to do that directly in production, because obviously without load this doesn't make any difference.

However, I'm a bit worried about this, because InnoDB actually has potential to be slower, so the question is:

If I convert all tables to InnoDB and it turns out i'm worse off than before, can I go back to MyISAM without losing anything?

Can you think of any problems I might encounter? (For example, I know that InnoDB stores all data in ONE big file that only gets bigger, can this be a problem?)

Thank you very much
Daniel

I'm trying to import a MySQL dump file, which I got from my hosting company, into my Windows dev machine, and i'm running into problems.

I'm importing this from the command line, and i'm getting a very weird error:

ERROR 2005 (HY000) at line 3118: Unknown MySQL server host '╖?*á±dÆ╦N╪Æ·h^ye"π╩i╪ Z+-$▼₧╬Y.∞┌|↕╘l∞/l╞⌂î7æ▌X█XE.ºΓ[ ;╦ï♣éµ♂º╜┤║].♂┐φ9dë╟█'╕ÿG∟═0à¡úè♦╥↑ù♣♦¥'╔NÑ' (11004)

I'm attaching the screenshot because i'm assuming the binary data will get lost...

I'm not exactly sure what the problem is, but two potential issues are the size of the file (2 Gb) which is not insanely large, but it's not trivially small either, and the other is the fact that many of these tables have JPG images in them (which is why the file is 2Gb large, for the most part).
Also, the dump was taken in a Linux machine and I'm importing this into Windows, not sure if that could add to the problems (I understand it shouldn't)

Now, that binary garbage is why I think the images in the file might be a problem, but i've been able to import similar dumps from the same hosting company in the past, so i'm not sure what might be the issue.

Also, trying to look into this file (and line 3118 in particular) is kind of impossible given its size (i'm not really handy with Linux command line tools like grep, sed, etc).

The file might be corrupted, but i'm not exactly sure how to check it. What I downloaded was a .gz file, which I "tested" with WinRar and it says it looks OK (i'm assuming gz has some kind of CRC). If you can think of a better way to test it, I'd love to try that.

Any ideas what could be going on / how to get past this error?

I'm not very attached to the data in particular, since I just want this as a copy for dev, so if I have to lose a few records, i'm fine with that, as long as the schema remains perfectly sound.

Thanks!
Daniel

I'm having a performance problem in a site we've made, and I'm not exactly sure how to start diagnosing it.

The short description is: We have a very small site (http://hearablog.com) with very little traffic, in a crappy dedicated server, CPU is always very high, sometimes it stays at 100% for minutes, and w3wp.exe is taking most of it. A typical scenario is w3wp.exe takes 60%, and SQL Server takes about 30%. Our DB is pretty small too.

Long description and more details:

• The site is hosted in a very crappy server by Cari.Net. From the beginning we had the feeling that the server didn't quite behave correctly, like some things would take just too long, so this could be a configuration problem from the get go. It may also be that we are getting a virtual server while we're supposed to have a dedicated one, although we have no evidence that'd indicate this, except for the fact that the server tends to be quite slow.

• The server is Windows 2008 Standard 64-bit, with SQL 2008 Express

• Hardware is a Celeron 2.80 GHz, 1Gb RAM

• The website is developed in ASP.Net MVC, using Entity Framework for data access.

• Now, this is pretty crappy hardware, but i've had other servers with these guys, with equivalent (or worse) HW, and performance is much better than this one. That said, the other servers have W2003 and SQL2005, and I'm using ASP.Net "WebForms" 2.0, no MVC, no LINQ, no EF; so I'm not sure whether going to 2008 / the other stuff means a big performance penalty is expected.

• I'm serving MP3 files (5-20 Mb) regularly, which is a slightly unusual load, maybe that is causing some kind of problems?
  Would that cause w3wp to use a lot of CPU?

• Disk usage seems very low. Memory is usually around 90%, but disk usage seems to indicate it's not paging much.

• I get tons of e-mails every day about SQL timeouts, for queries taking over 30 seconds, although all our queries are pretty straightforward (or should be, but EF may be screwing it up).

This is what resource monitor looks like in one of these "sprints" of 100% CPU, in case there's anything useful there.

And a snapshot of some performance counters:

Now, what confuses me very much is that CPU usage of w3wp is just so high. It shouldn't be doing much really... So my questions are...

• Is there any way of finding out "what" it is doing? Maybe even profile it?
• Any performance counters I should be looking at?
• Is this to be expected given this hardware/software configuration?
• Is this could be cause by some kind of configuration failure, where would you start looking?

Thank you VERY much.
Daniel Magliola

NOTE: Although there's some PHP code below, I really believe this belongs to ServerFault more than StackOverflow, esentially, because this worked and it stopped working after we changed servers.

I have the following situation:

• There's 2 public websites with different functionalities (UPDATE: both websites are on the same physical server). A user can fill up a huge signup form in Website A to get registered to website B.
• When a user registers in Website A, I send an Ajax request to a PHP script in Website A to validate the data.
• This script is esentially a proxy that sends all that data to Website B, and B responds with "OK", or with a bunch of validation errors. It only exists because I can't do cross-domain Ajax directly.
• All these are HTTP POSTs over SSL

This was working beautifully, until we migrated our server, in the same hosting company, to a bigger one. They hosting people took care of all the migration, and the idea was that the new server would end up with the same configuration as the old one.

Now, after this migration, when the information submitted to the proxy, converted to an "HTTP POST string", takes more than 1024 bytes, the server-side request from Website A to B fails silently. It works, it executes all the lines of code, there are no errors, notices or exceptions that I can see. It just goes all the way to the end, as if website B had responded with an empty string.

If, however, the POST is less than 1024 bytes, it all works great, and I either get the correct validation errors, or "OK" back.

Sending the full POST to Website B directly, without the proxy, works correctly, so the problem is definitely on the proxy call.

I have no rationalization for this behaviour, this makes absolutely no sense to me.

Has any of you ever encountered something like this before?
Any pointers on where I could look?

For reference, this is the proxy code:

$CustomersValidateURL = "https://WebsiteB.com/validatescript";
echo do_post_request($CustomersValidateURL, http_build_query($_POST));

// =========================================================================
function do_post_request($url, $data, $optional_headers = null){

    $params = array('http' => array('method' => 'POST', 'content' => $data));

    if ($optional_headers !== null){
        $params['http']['header'] = $optional_headers;
    }

    $ctx = stream_context_create($params);
    $fp = @fopen($url, 'rb', false, $ctx);

    if (!$fp){
        throw new Exception("Problem with $url, $php_errormsg");
    }

    $response = @stream_get_contents($fp);

    if ($response === false) {
        throw new Exception("Problem reading data from $url, $php_errormsg");
    }

    return $response;
}

This is the result of running a tracepath from that server to "Website B", in the same server:

tracepath customers.xxx.com
 1:  server.xxx.com (208.92.xxx.xxx)      0.144ms pmtu 16436
 1:  server.xxx.com (208.92.xxx.xxx)      0.093ms reached
     Resume: pmtu 16436 hops 1 back 1

I have the following situation: I have a production site running on SQL 2005 Express (only because it's free), and it's a reasonably big DB, and i'm always incredibly close to the 4 Gb limit. Every now and then I actually hit it and it's a big mess.

Now that SQL 2008 Express has 10Gb as a limit, I'm going to migrate.

The question is, if I just install SQL 2008, and attach the MDF file from 2005, will I have the 10 Gb limit? Or do I have to create a new Database from 2008, and then import the data?

If the latter, how do you recommend to do this import, what's the fastest and simplest way?

Thank you very much!
Daniel

I have a website that is sending out "cache-control" and "pragma" HTTP headers for PHP requests.
I'm not doing that in the code, so I'm assuming it's some kind of Apache configuration, as suggested by this question (you don't really need to go there for this question's context)

I don't have anything in my .htaccess files, so it's gotta be in Apache's configuration itself, but I can't access that, this is a shared hosting, I only have FTP access to my website's directory.

Is there any way that I can add directives to my .htaccess files that will remove the headers added by the global configuration, or otherwise override the directive so that they're not added in the first place?

Thank you very much
Daniel

sorry if this is a very dumb question, but...

I just got an offer from a dedicated hosting company, offering me a box with "2 Core 2 Quad processors @ 6GHz"

I didn't know we could get 6GHz, and some googling turned out a bunch of people getting that, after overclocking like crazy, which seems to confirm my suspicion.

Are there hostings offering 6GHz CPUs, or are these people outright lying to me?

Thank you!

I'm running a server with SQL 2008, and backing up the database with a "regular maintenance plan". (I'm not an expert by far in configuring SQL Server).
The execution of this plan is automated by a SQL Server Agent Job.

This seems to work, I have my backup files in the hard drive, and every now and then I download them and restore them in my machine and they seem to work.

However, I tried to run the backup manually today before making some big changes to the schema, and it reported it as failed. However, the backup file did end up in the hard drive, and I was able to restore it in my machine too.

So, I went looking for the history of execution of the Agent job, and sure enough, it's been failing every day. However, I have no idea what is wrong. Again, the backups seem to work, but the fact that it's reporting them as failures every day is a little unsettling for me.

This is the best log I could find of the problem, but I have no idea what to make of it.
Any ideas / pointers on where I should start looking?

Date        1/24/2010 9:28:55 AM
Log     Job History (Backup.Backup)

Step ID     1
Server      MyServerName
Job Name        Backup.Backup
Step Name       Backup
Duration        00:00:09
Sql Severity        0
Sql Message ID      0
Operator Emailed        
Operator Net sent       
Operator Paged      
Retries Attempted       0

Message

Executed as user: KVARDEKDU\MyServerName$. 
Microsoft (R) SQL Server Execute Package Utility  Version 10.0.2531.0 for 64-bit  
Copyright (C) Microsoft Corp 1984-2005. All rights reserved.    
Started:  9:28:55 AM  
Progress: 2010-01-24 09:28:58.85     
Source: {8C5D2693-8EF1-4505-95A0-735311CBA89C}      
Executing query "DECLARE @Guid UNIQUEIDENTIFIER      
EXECUTE msdb..sp...".: 100% complete  
End Progress  Progress: 2010-01-24 09:29:00.26     

Source: Back Up Database Task      
Executing query "EXECUTE master.dbo.xp_create_subdir N'E:\Backups\\...".: 100% complete  
End Progress  Progress: 2010-01-24 09:29:03.07     

Source: Back Up Database Task      
Executing query "BACKUP DATABASE [MyDBName] TO  DISK = N'E:\Backup...".: 50% complete  
End Progress  Progress: 2010-01-24 09:29:03.90     

Source: Back Up Database Task      
Executing query "declare @backupSetId as int  select @backupSetId =...".: 100% complete  
End Progress  DTExec: The package execution returned DTSER_FAILURE (1).  

Started:  9:28:55 AM  
Finished: 9:29:04 AM  
Elapsed:  9.047 seconds. 
 The package execution failed.  The step failed.

Thanks!
Daniel

I have a process I run from a batch file, and i only want to run it on a certain day of the week.
Is it possible to get the day of week?

All the example I found, somehow rely on "date /t" to return "Friday, 12/11/2009", however, in my machine, "date /t" returns "12/11/2009". No weekday there.

I've already checked the "regional settings" for my machine, and the long date format does include the weekday. The short date format doesn't, but i'd really rather not change that, since it'll affect a bunch of stuff I do.

I have a network where my WAN (a CableModem) goes into a (dubiously configured) Ubuntu box which acts as the router for the whole company.

We recently switched our ISP, and we're getting a very weird behavior. In Vista/Server 2008 machines, some websites don't work. This comes and goes a bit, but there's one, www.dilbert.com that never works.

We've been experimenting with this over weeks, and these are the things we've found so far...

• The Linux server itself CAN load the Dilbert homepage
• Windows XP/2003 machines work PERFECTLY
• Windows Vista/2008 machines have problems
• Sometimes, on my machine (2008), in FireFox, the page loads. It never loads in Chrome, for example. Now, once it loaded in Firefox, if I press F5, it dies immediately, it only loads once. The error is "the connection was reset"
• Restarting my machine, the router, the cablemodem, etc, won't make it load again, I still get "connection reset", it also doesn't work from other machines. Some time afterwards (days), however, it will load again, once.
• Also, I can sometimes open a telnet connection (using the Windows telnet client) to www.dilbert.com:80 and do a "get /", in which case I get a bunch of HTML, and the connection closes, expectedly. Now, if I open the same connection again, it connects, but as soon as I press "g", it disconnects me. This is the lowest level kind of information I could find.
• Using "Fiddler" to inspect the network traffic, with some browsers the request shows with "response status 0", an empty response, and no error details. With other browsers, it doesn't even appear.
• Finally, if I connect the cablemodem directly to a 2008 PC, without the router in between, everything works wonderfully, so I'm 99.99% sure the problem is inside my network, and not with my ISP.

Now, my best guess is that this is some kind of weird interaction between Vista's network layer (which as I understand, was rewritten from XP's, it's not the same), and Ubuntu's.
That's all I got. Besides that, I'm completely dumbfounded, and starting to believe there's a curse on my building.

Can any of you think of any plausible ideas as to what might be going on / what I could do to fix / diagnose this?

NOTE: I know NOTHING about linux, although I can run commands if you can think of something that'll give me some useful info.

Thanks!
Daniel

I'm stress testing a site that we're making, and we're finding a very surprising result compared to my expectations:
Our site starts to load very slow with a few hundred simultaneous people, even though CPU and memory are fine. Looking at Task Manager, Networking tab, I see that my 100 Mbps network card is maxed out at 98%.

For some reason, this sounds extremely ridiculous to me...
Every time I read something on scalability it's CPU, Memory, Caching, etc, etc, and here I'm getting the bottleneck on the network card itself.

We serve all our content gzipped, and our home page is kind of heavy, but not THAT much. I would've never expected the network card to be the bottleneck.

Is this normal?
Is everyone having public facing websites using a 1Gbps network card?
I thought 100 Mbps would be the standard.

Am I looking at something wrong? Am I interpreting the graph in the Networking tab incorrectly?

NOTE: I can think of a number of ways to fix this, starting with getting a 1 Gbps card, and moving static files to their own server(s). My question is mostly around whether everyone is simply using 1 Gbps connections, which would surprise me enormously.

I'm running a server with 3 database instances.
Right now, when I open Management Studio, it opens the "Connect" prompt that lets me choose which instance I want (or connect to anything else).

Would it be possible to have it so that, when I open it, it's already connected to these 3 instances, so I can start working right away and save a few clicks? It'd save me hundreds of clicks a day, honestly.

This would be equivalent to what we had in SQL 2000 Enterprise Manager, where you could "register" servers, and they'd always be there just a click away.

EDIT: To clarify what I'm looking for... Before we had Enterprise Manager and Query Analyzer. Now SSMS is the blend between those two, but the "connection" part works exactly like Query Analyzer.
What I do every day is open SSMS to manage the DBs, check their stats, rebuild an index here and there, I'm not using it to run that many queries, so I'm looking for a way to have the SQL instances available on the Object Explorer automatically when logging in. Is this impossible now? :-(

I'm trying to get Windows Server 2008 x64 with Hyper-V in my machine.

Do I need to get an edition of Windows that has Hyper-V?
Or can I install windows without Hyper-V and then install "Hyper-V" as a separate product?

(I want the regular Windows with a GUI, not the "core" variant)

Thanks!

I have a user that needs to be able to access the Performance counters tool in one of my servers, but I can't make him an admin.

Is there a way to grant him this permission?

Thanks!
Daniel

Could anyone here explain to me what is implied by this term? (I've seen the same thing mentioned with the 3 terms).

At first when I read about it, for some reason I understood that it was some way of splitting the bytes across the platters of the disk, which sounded like a good idea and obviously doesn't make sense, because that wouldn't cut disk size in half (and disk are probably already splitting bytes across platters)...

The best I've come to understand is that basically instead of creating one partition for the whole size of the disk, you create 2 partitions, and use only one of them, either the one in the "center" or the one in the "rim" of the platters, and since one of the two is faster (people didn't seem to agree on which one was faster), that makes everything better.

Am I understanding this correctly?
Has anyone tried this with their drives and had a good outcome?

Thanks!

I have just restarted my dev machine, and now MySQL refuses to start up. When I start the service, it instantly throws an exception and dies...

I can't think of anything I could've changed before restarting that could've affected MySQL in any way...

This is MySQL Server 5.1 running on Windows 2003 Server.

This is what the log file says:

090703 14:11:17 - mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=26214400
read_buffer_size=65536
max_used_connections=0
max_threads=100
threads_connected=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 58231 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
006B8853    mysqld.exe!srv_parse_data_file_paths_and_sizes()[srv0start.c:211]
006A7E18    mysqld.exe!innobase_init()[ha_innodb.cc:1536]
00442748    mysqld.exe!ha_initialize_handlerton()[handler.cc:434]
00561253    mysqld.exe!plugin_initialize()[sql_plugin.cc:1002]
00565AE5    mysqld.exe!plugin_init()[sql_plugin.cc:1209]
004CE1B4    mysqld.exe!init_server_components()[mysqld.cc:3831]
004CE8C6    mysqld.exe!win_main()[mysqld.cc:4267]
004CECDB    mysqld.exe!mysql_service()[mysqld.cc:4439]
00724593    mysqld.exe!_callthreadstart()[thread.c:293]
0072462C    mysqld.exe!_threadstart()[thread.c:275]
77E6482F    kernel32.dll!GetModuleHandleA()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Any ideas?
Should I uninstall and reinstall MySQL?
Is there anything quicker than I can do?

Thanks!