SunSparc's questions

In my "/etc/haproxy/haproxy.cfg" file, I specify a whitelist file that contains IP addresses that are permitted to access a frontend.

frontend default-frontend
    <snip>

    tcp-request connection reject if ! { src -f /etc/haproxy/templates/ip-whitelist.txt }

    <snip>

The contents of "/etc/haproxy/templates/ip-whitelist.txt" look like this:

192.45.21.89/32
123.34.33.7/32
56.23.12.77/32
78.12.66.3/32

This works great! Until I want to clean up the file and remove IPs for people who no longer need access.

Question: Is it possible to add comments to an haproxy template file?

I have tried this:

192.45.21.89/32 # Dylan Reeve
123.34.33.7/32 # Jane Doe
56.23.12.77/32 # Priscilla Ahmed
78.12.66.3/32 # Sayed Salas

... which returns errors similar to:

[ALERT] : parsing [/etc/haproxy/haproxy.cfg:123] : 'tcp-request connection reject' :
    error detected in frontend 'default-frontend' while parsing 'if' condition : 
    '192.45.21.89/32 # Dylan Reeve' is not a valid IPv4 or IPv6 address
    at line 1 of file '/etc/haproxy/templates/ip-whitelist.txt'
[ALERT] : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] : Fatal errors found in configuration.

Worst case scenario: I have to keep separate list in another location to match IPs with names.

I am running Docker on Ubuntu Linux. I have a Dockerfile that brings in a lot of data from elsewhere. I put the Dockerfile on an external drive that I have mounted to the local filesystem (ext4). Then I ran the build from within that external drive filesystem.

docker build -t mycontainer .

I watched the drive usage as the container was building. All of the drive usage was on the root filesystem instead of the attached drive.

I have read some articles on changing the Docker root/base from /var/lib/docker to another location, but so far all I get is docker.service: Failed with result 'exit-code'.

I also tried adding the -g options in the /lib/systemd/system/docker.service file:

ExecStart=/usr/bin/dockerd -H fd://  # original
ExecStart=/usr/bin/dockerd -g /new/path/docker -H fd://  # updated

This did not help either.

How can I tell Docker to use the external drive?

I have a webpage (https://smartystreets.com/contact) that uses jQuery to load some SVG files from S3 through the CloudFront CDN.

In Chrome I will open an Incognito window as well as the console. Then I will load the page. As the page loads, I will typically get 6 to 8 messages in the console that look similar to this:

XMLHttpRequest cannot load 
https://d79i1fxsrar4t.cloudfront.net/assets/img/feature-icons/documentation.08e71af6.svg.
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'https://smartystreets.com' is therefore not allowed access.

If I do a standard reload of the page, even multiple time, I continue to get the same errors. If I do Command+Shift+R then most, and sometimes all, of the images will load without the XMLHttpRequest error.

Sometimes even after the images have loaded, I will refresh and one or more of the images will not load and return that XMLHttpRequest error again.

I have checked, changed, and re-checked the settings on S3 and Cloudfront. In S3 my CORS configuration looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedOrigin>http://*</AllowedOrigin>
    <AllowedOrigin>https://*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>

(Note: initially had only <AllowedOrigin>*</AllowedOrigin>, same problem.)

In CloudFront the distribution behavior is set to allow the HTTP Methods: GET, HEAD, OPTIONS. Cached methods are the same. Forward Headers is set to "Whitelist" and that whitelist includes, "Access-Control-Request-Headers, Access-Control-Request-Method, Origin".

The fact that it works after a cache-less browser reload seems to indicate that all is well on the S3/CloudFront side, else why would the content be delivered. But then why would the content not be delivered on the initial page-view?

I am working in Google Chrome on macOS. Firefox has no problem getting the files every time. Opera NEVER gets the files. Safari will pick up the images after several refreshes.

Using curl I do not get any problems:

curl -I -H 'Origin: smartystreets.com' https://d79i1fxsrar4t.cloudfront.net/assets/img/phone-icon-outline.dc7e4079.svg

HTTP/1.1 200 OK
Content-Type: image/svg+xml
Content-Length: 508
Connection: keep-alive
Date: Tue, 20 Jun 2017 17:35:57 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Access-Control-Max-Age: 3000
Last-Modified: Thu, 15 Jun 2017 16:02:19 GMT
ETag: "dc7e4079f937e83291f2174853adb564"
Cache-Control: max-age=31536000
Expires: Wed, 01 Jan 2020 23:59:59 GMT
Accept-Ranges: bytes
Server: AmazonS3
Vary: Origin,Access-Control-Request-Headers,Access-Control-Request-Method
Age: 4373
X-Cache: Hit from cloudfront
Via: 1.1 09fc52f58485a5da8e63d1ea27596895.cloudfront.net (CloudFront)
X-Amz-Cf-Id: wxn_m9meR6yPoyyvj1R7x83pBDPJy1nT7kdMv1aMwXVtHCunT9OC9g==

Some have suggested that I delete the CloudFront distribution and recreate it. Seems like a rather harsh and inconvenient fix.

What is causing this problem?

Update:

Adding response headers from an image that failed to load.

age:1709
cache-control:max-age=31536000
content-encoding:gzip
content-type:image/svg+xml
date:Tue, 20 Jun 2017 17:27:17 GMT
expires:2020-01-01T23:59:59.999Z
last-modified:Tue, 11 Apr 2017 18:17:41 GMT
server:AmazonS3
status:200
vary:Accept-Encoding
via:1.1 022c901b294fedd7074704d46fce9819.cloudfront.net (CloudFront)
x-amz-cf-id:i0PfeopzJdwhPAKoHpbCTUj1JOMXv4TaBgo7wrQ3TW9Kq_4Bx0k_pQ==
x-cache:Hit from cloudfront

Today I created an AutoScaling Group with several policies, each with an Alarm associated with a CloudWatch metric. The AutoScaling Group launched two instances, because that the minimum that I required for that group. I turned on my benchmarking tests to hit the new group and their combine average CPUUtilization rose above the thresholds that I had set for the Alarms. I watched the History for each of those Alarms as the states changed from INSUFFICIENT_DATA, then to OK, then finally to ALARM. As expected. Then I waited for the Policies to be executed. The specified 300 seconds before execution came and went. Then I decided to see how long it would take before the Policies executed. It has now been over an hour and a half and those Policies have yet to execute.

I got frustrated and deleted the ASG and all the Policies and Alarms. I decided that perhaps I had made an error in creating the Policies and Alarms. So I recreated them all. Same problem. The Alarms are in ALARM state and never execute the Policies. I can execute the Policies manually without problem, but that defeats the point.

If it makes any difference, I created all these things in the AWS Console. In the past I have used the API to create ASGs and Policies and those groups execute policies just fine.

Is there something I am doing wrong? (Or perhaps AWS services are just having issues.)

UPDATE:

Tried this again today. Decided to use the API instead of the Console to see if there was any difference. The Alarm I am using just updated from OK to ALARM 20 minutes ago.

CPUUtilization >= 35 for 5 minutes
Actions: In ALARM: For group TEST use policy Scale Out API (Add 1 instance)
Namespace: AWS/EC2
Metric Name: CPUUtilization
Dimensions: AutoScalingGroupName = TEST
Statistic: Average
Period: 5 minutes

2013-12-19 15:13 UTC-7 - State update - Alarm updated from OK to ALARM

Current time: 15:43 UTC-7 (30 minutes since alarm)

Action has not triggered. Any idea why?

Problem: I need to copy a large block of data from a remote location into system memory as quickly as possible.

Scenario: I have a data processing system. The system is built via shell scripts on-the-fly using multiple components that are pulled in from remote locations.

One of those components is a large block of data stored as groups of files.

The requirement I have is to retrieve that large block of data from a remote location and install it into system memory as quickly as possible. This is a requirement so that the system which relies on this data can start using it for processing as soon after boot time as possible.

Question: "What would be the most efficient container for my data?"

Solutions already tried/considered:

• ISO file: requires tools for creation and reading that are not typically native
• TAR file: extracting can take a lot of time
• Remote filesystem mounted as local: slow because contents need to be copied into memory
• LVM snapshot: gear more toward backups, not built for speed on restore

Notes:

• Data loss is not a primary concern.
• The remote file transfer procedure is not a primary concern as I already have an adequate tool.
• The system is currently using Ubuntu Linux.

I have a web app (oldapp.com) that stores images in a sub-folder (oldapp.com/images) of the DocumentRoot (aka web root). I am rebuilding the app on a different domain (newapp.com) and have created a sub-domain specifically for storing images (images.newapp.com). While I am building, testing, and using the new app, I still need the old app to continue accessing that old image sub-folder. All of this is being done on the same server. I need the new sub-domain to have the same access to the images that are in the old sub-folder.

For this particular case, I do not have the option of changing the DocumentRoot directive. I do have access to Alias and Redirect.

Can I Alias the DocumentRoot of the images.newapp.com vhost to the old sub-directory of another vhost (/var/www/vhost/oldapp.com/httpdocs/images)?

In other words, something like this in the images.newapp.com vhost block:

AliasMatch ^/(.*) /var/www/vhost/oldapp.com/httpdocs/images$1