[EDITED to add: This problem has gone away on its own. I believe Cloudflare's name resolution may have been to blame. See my own answer below]
Here is a snippet of my zone file
*.example.com. 300 IN CNAME proxy.herokuapp.com.
foo.example.com. 300 IN A 111.111.111.111
If I dig @8.8.8.8 foo.example.com
I get the answer I expect:
;; ANSWER SECTION:
foo.example.com. 30 IN A 111.111.111.111
The same is true of all other public DNS servers I've tried.
However, when I try to set up a check with Pingdom to a URL on foo.example.com
it instead sends the traffic to my Heroku app referenced by the *.example.com
RR.
The same is true of checks set up on New Relic, Errplane and traffic generated by the Heroku app itself.
So on the one side, all public DNS servers interpret the zone file one way. Yet four service providers all interpret it a different way, one that differs to the standard suggested by RFC 4592.
My question is: are these reputable, mature service providers all wrong? Or is it little me?
From a protocol standpoint, the authoritative nameserver synthesizes responses to queries that fall within a wildcard, not the servers that query it. If you are getting different responses, then one of the following has to be the case:
After around 10 days the problem went away on its own. My best guess is that Cloudflare (who run the nameservers for this domain) changed its DNS resolution code.
Some more information:
Andrew B's answer (2) is a candidate: the name was being resolved by caching servers that hadn't been refreshed since I added the A-record. The problem persisted for more than a week so I don't believe this is likely.
I raised the issue with Cloudflare and it was escalated to the engineering team. They recently released some updated name resolution code (https://twitter.com/eastdakota/status/351733117003902976)
I think I understand how name resolution works a bit better than when I asked the question. It seems more likely that there was a single cause (Cloudflare bug) than multiple causes (separate bugs at Heroku, NewRelic, Pingdom & Errplane).