Oliver Henriot's questions

How do you account for GPU compute time on your HPC clusters ?

I have a growing, and quite heterogeneous (SXM4 A100s, PCIe A100s, NVlinked V100s, PCIe V100s, T4s, AMD cards arriving soon etc...), GPU partition on an HPC cluster (mixed hardware Debian servers running OAR scheduler).

Traditionally, we accounted compute time as seconds per core per job. Despite CPU and memory variability between nodes (fat nodes, high speed nodes, standard nodes), the difference was sufficiently small that it didn't impact accounting noticeably, especially in a small university setting.

On GPUs, things change quite a bit. The difference in performance and cost between an SXM4 A100 node and a T4 are quite significant and our current model is probably not going to cut it, moreover as growing university partnerships impose that we host more and more private sector projects which we will have to account for precisely.

I'm exploring how to do this accounting with our current infrastructure but was also wondering what methods by other people operating HPC GPU clusters. If you have any advice as to how to do this or what strategy/tools you have used, I'd be very willing to hear them!

Thanks!

I've already asked the question on the Docker forums but so far no luck with a reply. I'm hoping I will have better fortune here.

I’m in the process of making an ssh bastion (with fail2ban for security) in a Docker container. The container will serve no other purpose than being an ssh bastion and will serve to access bound volumes via sshfs.

Currently, I’m using what I suspect is a very poor method. My Dockerfile:

• pulls the latest debian stable-slim image,
• installs ssh and fail2ban via apt,
• adds users, sshd_config and fail2ban jails,
• then specifies a wrapper bash script in which I unconditionally start ssh and fail2ban with custom configuration paths in /app

It works, with some caveats (no /var/log/auth.log for fail2ban to eat so quite a bit of configuration twisting there), but I feel I’m doing it wrong.

The problem is it’s super hard to find information regarding how to properly do what I’m trying to do. There are endless pages listing how to ssh into a container for debugging, backup etc. There are also some interesting pages about why you shouldn’t ssh into a container. But I haven’t found the holy grail page of “this is how you properly setup an ssh bastion with fail2ban in a Docker container”. Does it even exist? Or am I right to do it the way I am?

Thanks!

I have an openldap server (with user passwords) open worldwide which I'm trying to secure.

Step 1 was to limit access to data to authenticated users via ACLs.

Step 2, to prevent brute force attacks, was to implement ppolicy. Seems to be working fine, cool.

Step 3 is going to be "handle the user who's been locked out and swears it's not his fault" by spotting dn lockouts as early as possible with their possible causes.

I've started writing scripts which check pwdAccountLockedTime attribute presence, warn via email, ring bells etc. That's fine but I'm finding hard to link that to data in the logs saying when the incriminated logins occurred, where they were done from etc. All the data is there but bringing it all together is a real pain. I feel sure I'm not the only one confronted with this problem (or am I trying to solve the wrong problem ?) and that solutions exist, I just haven't been able to find them. Am I wrong ?

Forgot to say, fail2ban is not really suited. There are many clients, of whom I do not necessarily know the addresses, who are likely to do legitimate massive requests on the directory and would not pass fail2ban. Sounds strange, I know, but our configuration here is complicated and we have to make do with it. That's why I'm looking at ppolicy.

To put it short, I'd like to have a way to monitor the occurrence of pwdAccountLockedTime and, when that happens, immediately have the info of which user is concerned, the pwdFailureTime values, what requests were done at that time and from which IP address(s) in a single, easy to read, log file. That would be great, surely it exists ?

I was wondering whether JtR is a good solution to test password strength. I'm not interested in actually finding the passwords (I'd rather not in fact), what I'd like is to run JtR in a way which would allow me to say "User A, I think you ought to choose a better password, it's really rather weak; user B, your password is decent but could be improved a little; user C, well done, your password is strong" in order to do a bit of user education. Is there a way to get JtR to only say how long it's been running when it's found a match rather than record the password in the .pot file? Is there a better way to do this? And no, before you say it, I know it'd be perfect if I could just say, passwords should respect such standards but that's unfortunately not an option currently.

Thanks

I have an openldap 2.4 server on centos 6 whose confg I copied from those I have running under openldap 2.3 servers on centos 5 machines.

On openldap 2.3, specifying TLSCACertificateFile, TLSCertificateFile and TLSCertificateKeyFile with correct values makes the server listen on port 636. This is not the case on the openldap 2.4 setup.

I have configured it with loglevel -1 but I have not seen any clue as to what might be wrong and reading the openldap 2.4 manual doesn't indicate if any of the other TLS related parameters are now mandatory. I don't think so though because if I run the service manually, using "# /usr/sbin/slapd -u ldap -h "ldap:/// ldaps:/// ldapi:///"", the server does listen on port 636 and I can query it using "ldapsearch -H ldaps://myserver:636".

Is there something I am missing to get the server to listen on port 636 without having to always launch it manually? Is this linked to centos 6 or openldap 2.4?

Thank you.

Cheers,

I need to store account creation and expiry dates in our existing ldap. None of the standard existing schemas seem to quite fit the bill, not even ppolicy or samba.

The answer I've come across so far is: "use a custom schema".

Fine, but I can't seem to find an existing custom schema to hold account creation and expiry dates. Shurely someone has already had to store this kind of info in their ldap and has therefore used (maybe even created) such a schema, no? If I can't find one, does this mean I will really have to write my own custom schema and all? Seems like I'd be inventing the wheel again frankly, I'm sure I've missed something but I just can't find an existing schema for my needs (or at least one from which I could take existing attributes and objectClasses and plonk them in my schema...).

Am I missing something obvious here?

Thanks.

I have a bind server which, when you query it this way (from localhost, obviously) :

host nomachinehasthisname 127.0.0.1

replies :

;; connection timed out; no servers could be reached

It throws the same error if I query it using its public address.

It throws the same error no matter from which client I do this query.

It does not throw the error if I use a fqdn as the query value.

It throws the error only if the query is on a A record not present in my dns.

If I query a non-existant A record using a fqdn query replies NXDOMAIN as expected.

If I query it using dig +trace :

dig +trace nomachinehasthisname @127.0.0.1

it shows that my server tries to query the root servers and then times out (which is logical).

The odd thing is that I have 4 other bind servers with pretty much identical configurations and none of them have this behavior. I can't figure what it is that is missing on this server which is making it fail to reply the same way as the other servers.

I have these options in my named.conf :

options {
        directory "/var/spool/named";
        auth-nxdomain no;
        allow-recursion { internal_networks; };    // restrict recursion
};

and it's more or less the same for the other servers.

I would like my server to default to the search domain listed in /etc/resolv.conf for non-fqdn A record queries, which is what the other servers do.

The resolv.conf is similar between this server and the others. I even have some servers which have less info (no "search domain" info for example) and do not have this problem.

How do you get bind to default to a particular domain for non-fqdn A record queries?