T. Brian Jones's questions

I have setup an SFTP server using OpenSSH. The home directory for users is /sftp/%user. I'm mounting an S3 bucket at /sftp using S3FS. The problem is that S3FS cascades user permissions down through it's directory structure, meaning:

1. I can configure files at /sftp/* to have these permissions drwxr-xr-x 1 root root which allow SFTP users to connect, but they cannot write to their home directories because they don't own them.

s3fs nwd-sftp /sftp/ -o iam_role=sftp-server -o allow_other -o stat_cache_expire=10 -o enable_noobj_cache -o enable_content_md5 -o umask=022

2. I can configure files at /sftp/* to have permissions drwxrwxr-x 1 root sftpusers so users can (in theory) write to their home directories, but the SSH protocol won't let them login because it considers these permissions incorrect (allowing members of a group write access).

s3fs nwd-sftp /sftp/ -o iam_role=sftp-server -o allow_other -o stat_cache_expire=10 -o enable_noobj_cache -o enable_content_md5 -o umask=002 -o gid=501

I can't figure out how to customize the ownership or permissions once a drive is mounted with S3FS. Is there a way to do this? How can I customize the users' home folders within the S3 mounted /sftp folder?

I have setup an SFTP server using OpenSSH. The home directory for users is /sftp/%user. I'm mounting an S3 bucket at /sftp using S3FS. The problem is that S3FS cascades user permissions down through it's directory structure, meaning:

1. I can configure files at /sftp/* to have these permissions drwxr-xr-x 1 root root which allow SFTP users to connect, but they cannot write to their home directories because they don't own them.

s3fs nwd-sftp /sftp/ -o iam_role=sftp-server -o allow_other -o stat_cache_expire=10 -o enable_noobj_cache -o enable_content_md5 -o umask=022

2. I can configure files at /sftp/* to have permissions drwxrwxr-x 1 root sftpusers so they can (in theory) write to their home directories, but the SSH protocol won't let them login because it considers these permissions incorrect (allowing members of a group write access). I can't assign ownership on a per user basis with S3FS.

s3fs nwd-sftp /sftp/ -o iam_role=sftp-server -o allow_other -o stat_cache_expire=10 -o enable_noobj_cache -o enable_content_md5 -o umask=002 -o gid=501

Is there a solution I'm overlooking with the OpenSSH SFTP configuration that would allow users to login and write to their directories? They are CHRooted to their home directory, so I see no reason I can't give read/write permissions to all SFTP user home directories for the group sftpusers that they all share.

Is it possible to bypass SSH's issues here? What security flaws am I exposing in doing so?

I have installed ProFTPd on an Amazon EC2 running Amazon Linux.

I have enabled SSL (FTPS) on for ProFTPd and set passive ports in proftpd.conf:

port                         21

<IfModule mod_tls.c>
  TLSEngine                  on
  TLSLog                     /var/log/proftpd/tls.log
  TLSProtocol TLSv1.2
  TLSCipherSuite AES128+EECDH:AES128+EDH
  TLSOptions                 NoCertRequest AllowClientRenegotiations
  TLSRSACertificateFile      /etc/proftpd/ssl/proftpd.cert.pem
  TLSRSACertificateKeyFile   /etc/proftpd/ssl/proftpd.key.pem
  TLSVerifyClient            off
  TLSRequired                on
  RequireValidShell          no
</IfModule>

MasqueradeAddress            ip-of-my-server
PassivePorts                 60000 65535

Users can connect via port 21 if I open up TCP Ports 60000 through 65535 on the security group associated with this instance, but this doesn't feel secure to me (I'm not familiar with Passive FTP or opening up a range of ports like this). We whitelist the IPs of all our customers who will connect to this server on port 21.

Questions

1. Is there a way to open these passive ports without opening them in the security group opn AWS, perhaps using ip configurations on the EC2 server like in this article (http://www.proftpd.org/docs/howto/NAT.html)? I'm not familiar with these types of configurations and am not sure what to do here.

2. Since a user can't connect to the FTPS server without going through port 21 which is whitelisted to their IP, is it ok to have the TCP ports 60000-65535 open to all IPs within the AWS Security Group? What security concerns should I have here?

3. Is there some other "best practices" way to configure this?

How do I allow multiple SFTP Users with S3FS and OpenSSH?

Everything works, except SFTP Users don't have permission to write to their Chrooted Home Directory: remote open("/some_file"): Permission denied

Setup

I've got an Amazon EC2 instance running Amazon Linux. I've installed S3FS and mounted an S3 bucket. I've also configured OpenSSH to allow SFTP Users to access a Chrooted Home directory inside the mounted S3 Bucket /s3_mounted_folder/user_folder/. I've successfully used the SFTP connection on a non S3 mounted directory. I've successfully used the S3 bucket to create and download files from S3 as root on the EC2 instance via SSH. My SFTP users can successfully download files from their /s3_mounted_folder/user_folder/ directory. The problem is that the SFTP users cannot put files into the S3 mounted folder.

The Problem ... I think

I am only able to configure all folders (/s3_mounted_folder/ and /s3_mounted_folder/user_folder/)with the same user:group and same permissions, thus, I can't give the user access to write to his/her home directory (/s3_mounted_folder/user_folder/). If I mount the bucket with the user or group and give either write permissions, then OpenSSH SFTP won't let users connect because it believes the user permissions are misconfigured (example: drwxr-xr-x 1 root root vs. drwxrwxr-x 1 root usergroup).

S3FS Commands

Here are the two different commands to launch S3FS in these two modes (where user 501 and group 501 are the SFTP user and group):

root user permissions (drwxr-xr-x 1 root root): sudo s3fs nwd-sftp /sftp/ -o iam_role=sftp-server -o allow_other -o umask=022

sftp user permissions (drwxrwxr-x 1 root usergroup): sudo s3fs nwd-sftp /sftp/ -o iam_role=sftp-server -o allow_other -o umask=002 -o gid=501

In that second scenario, the user would theoretically be able to put files into their home directory via SFTP, but SFTP won't let them connect because their Chrooted home directory has write permissions for a group that isn't root.

I have an EC2 instance that is setup as an SFTP server using OpenSSH. It only allows connections on whitelisted IPs via TCP Port 22 (limited via an EC2 Security Group). I constantly have customers try to connect from other IPs that they have not been whitelisted. I'd like to track those attempted connections and the IPs they come from so I can help them figure out their IP addresses.

Is it possible to see these IP addresses from the server? Can I also see the connection attempt and get the SFTP username?

I set up a VPC using scenario 2 from the AWS Docs: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html

I've assigned an Elastic IP to an EC2 instance running in a Public Subnet. SSH works fine and I can access the website running on it. I cannot, however make outgoing HTTP or HTTPS requests ( I noticed this when trying to run yum update ).

I believe all my security settings are correct. Am I not able to make outgoing HTTP/HTTPS requests through the internet gateway? I specifically requested not to have a public IP assigned to this instance when created, because I knew I was going to assign an Elastic IP that gets liked to the website DNS. I have a NAT setup for instances in the Private Subnet, but I currently only have RDS instances running there, so I have not tested outgoing requests from there.

The security group for that EC2 instance has the following outbound rules:

The routes table for that subnet has the following settings:

The Network ACL has the following settings:

The default DHCP Option Set has the following settings:

domain-name = ec2.internal
domain-name-servers = AmazonProvidedDNS

The default /etc/resolv.conf settings are:

search ec2.internal
nameserver 10.0.0.2

The CIDR Blocks for the VPC and Subnets are as follows:

VPC: 10.0.0.0/16
Public Subnet: 10.0.0.0/24
Private DB Subnet in US East 1A: 10.0.1.0/24
Private DB Subnet in US East 1C: 10.0.2.0/24

I've got a MySQL table with a .MYD file of 44GB, and a .MYI file of 34GB. I'm running scripts from the command line that analyze the data in this table. MySQL is not maxing out the CPU, my memory IS maxed, and VMSTAT is reporting that data is being swapped to disk ... but I'm not terribly familiar with this utility and it's reports.

I am running a High Memory Double XL Amazon EC2 Instance. It has 34GB of RAM. I am not very familiar with MySQL tuning and the system is running with the default install on Linux CentOS.

Does the current machine have enough memory to deal with this table efficiently? What other configurations do I need to worry about? The largest system available to me has 68GB of memory. What exactly is MySQL loading into memory when it loads a table? I'd like to get a better sense for what is going on.

• EXT3 supports 32,000 subdirectories.
• EXT4 supports 64,000...
• I've read about people having millions of subdirectories on the XFS filesystem, but can't find an exact value referenced anywhere.

How many subdirectories does the XFS filesystem support?

I've got an EXT3 formatted drive on a Linux CentOS server. This is a web app data drive and contains a directory for every user account ( there are 25,000 users ). Each folder contains files that that user has uploaded. Overall, this drive has roughly 250GB of data on it.

Does structuring the drive with all these directories impact drive read/write performance? Does it impact some other performance aspect I'm not aware of?

Is there anything inherently wrong or bad with structuring things this way? Perhaps just the wrong choice of filesystem?

I've recently tried merging two data drives and realized that EXT3 is limited to 32,000 subdirectories. This got me wondering why. It seems silly that I built it this way, considering each file has a unique id that corresponds to an id in the database. Alas ...

I am copying 25,000 folders containing hundreds of files each from one hard drive to another on the same Linux server. The destination folder already contains 25,000 different folders full of their own files. I am using the following command:

cp -v -r /folder_to_copy_from/* /folder_to_copy_to/

The process completed about 20%, then all of a sudden, every time it tried to copy a folder it gave the message "too many links" and skipped all the files in that folder, moving on to the next folder. That folder was never created. CP printed this message for all the remaining folders.

I estimate there were ~30,000 folders in the folder_to_copy_to when it stopped working. I am using a EXT3 formatted drive with plenty of space to hold the files on Linux CentOS.

If there is a max number of subdirectories allowed in a directory, is there a way to increase it? Should I investigate different formatting standards? Any other thoughts on what is going on?

How many individually functioning EBS drives ( non-raided ) can I connect to a single Amazon EC2 server running Linux CentOS? I can't find a value in the EC2 or EBS manuals. Are there any other drive limitations I should be aware of with EBS?

I am running Linux on Amazon EC2 severs. I need to copy millions of files that total hundreds of gigabytes between two EC2 systems in the same availability zone. I don't need to sync directories, I just need to copy all the files in one directory over to an empty directory on the other machine.

What is the fastest way to do this? Has anyone seen or run performance tests?

rsync? scp? Should I zip them first? Should I detach the drive they are on and re-attach it to the machine I'm copying to, then copy them? Does transferring over the EC2's Private IP speed things up?

Any thoughts would be appreciated.

NOTE: Sorry this was unclear, but I'm copying data between two EC2 systems both in the same AWS availability zone.

MY SETUP ( example )

I am running Linux on an Amazon High CPU Extra Large EC2 instance with the following specs:

• 7 GB of memory
• 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
• 1690 GB of local instance storage
• 64-bit platform

I have two large MySQL databases running on the MyISAM storage engine. One is 2GB and the other is 500MB. I'd like to make sure MySQL is using as much RAM as it can / needs to maximize query speeds. I know there are a bunch of MySQL memory config options like key_buffer_size and MyISAM_sort_buffer_size, but I'm unfamiliar with optimizing these.

QUESTIONS

1. How does one check what memory MySQL is currently using on a Linux system?
2. How does one maximize / optimize MySQL memory usage?
3. Assuming my queries and schema are optimized, what other changes should I consider?

THE SETUP

I'm running Linux CentOS on an Amazon EC2 instance.

The MySQL data files are on an EBS Drive mounted at /data/ ( symlink - /var/lib/mysql >> /data/mysql ).

Everything works fine in this setup.

THE PROBLEM

I'm trying to move everything from this EBS drive to a new drive. I umounted the /data/ drive, and mounted it at /data2/. Then I mounted the new drive at /data/ and copied everything over to it from /data2/. Everything on the system works great, except MySQL. Every time I try to start the MySQL daemon ( /etc/init.d/mysqld start ) I get a MySQL Daemon failed to start error.

I'm running httpd on linux.

I have a folder (/data/) that is not in the apache web directory (/var/www/html/) that I would like users to be able to access from their browser. I don't want to move this folder.

How do I make files in this folder accessible to a web browser when the folder is outside the apache web folder?

I'm setting up a virtual host on a linux machine.

I've already made necessary DNS changes and they have propagated and are working correctly.

There are many other virtual hosts setup in my /etc/httpd/conf/httpd.conf file that are working correctly (someone else set all these up). My config for the new virtual host is this (where the real IP has been replaced with ##.###.###.###):

<VirtualHost ##.###.###.###:80>
    ServerName www.website.com
    ServerAlias website.com
    DocumentRoot /var/www/html/website.com
    ErrorLog logs/website.com-error_log
    CustomLog logs/website.com-access_log combined
</VirtualHost>
<Directory /var/www/html/website.com>
    Order deny,allow
    Allow from all
    AllowOverride All
</Directory>

All the files for the site are sitting in /var/www/html/website.com. When I visit website.com, the main site on this server is displayed rather than website.com.

What are some other configuration settings (anywhere) that could be doing this? There are other virtual hosts set up in an identical manner (from what I can tell) and they are working correctly.

I accidentally changed ownership of /usr/bin/sudo to my current user (i also did this for some other stuff in my /usr directory). I can't change any of them back because I need ownership of /usr/bin/sudo to be root to do so. I do not have root access because I'm on an Amazon EC2 instance running linux.

Here's what I did (foolishly I know):

sudo chown -R ec2-user.ec2-user /usr/

I've also hosed a ton of other stuff in the process, but I think it can all be solved if I can reset ownership of /usr/bin/sudo

Please help. I'm brand new to Linux admin and am doing everything from the command line.

I have my important data on a separate, mounted EBS, but I'm awful with server admin and it'll probably take me an entire day to setup a new instance, but am afraid that is my only option.