Siku

From ACENET
Jump to: navigation, search


Siku is a high-performance computer cluster installed in 2019 at Memorial University in St. John's, Newfoundland.

It is funded in large part by the Atlantic Canada Opportunities Agency (ACOA) with the intention of generating regional economic benefits through industry engagement, while recognizing the important work that ACENET does for academic research in the region.

Siku is only accessible to selected clients. Industrial researchers should write to info@ace-net.ca. Principal Investigators of academic research groups may use this access request form.

Addresses

Login node (ssh): siku.ace-net.ca
Globus endpoint: computecanada#dtn.siku.ace-net.ca
Data transfer node (rsync, scp, sftp,...): dtn.siku.ace-net.ca

Authentication and authorization

If you have been granted access to Siku and you have a Compute Canada account, you can log in using your Compute Canada username and password.

We encourage you to use a passphrase-protected SSH key pair for regular access to Siku.

Known issues

  • Multi-Processing using libverbs is not working as expected. MPI implementations, however, should work.
  • Directories are automatically created at first logon. This may produce a race condition that results in errors like the following:
Could not chdir to home directory /home/username: No such file or directory
/usr/bin/xauth:  error in locking authority file /home/username/.Xauthority
Lmod has detected the following error:  Unable to load module because of error when  evaluating modulefile: ...

Should this occur on first login, simply log out, wait a minute, and log back in again.

Similarities and differences with national GP clusters

Siku is designed from experience gained with the Compute Canada systems, Béluga, Cedar, Graham, and Niagara. Users familiar with those systems will find much familiar here.

  • The filesystem is similarly structured. See Storage and file management.
    • There is no "Nearline" archival filesystem.
  • The same scheduler is used, Slurm, although with simpler policies. See "Job Scheduling", below.
  • The same modules system provides access to the same list of available software.

Job scheduling

Tasks taking more than 10 CPU-minutes or 4 GB of RAM should not be run directly on a login node, but submitted to the job scheduler, Slurm.

Scheduling policies on Siku are different from the policies on Compute Canada systems. What resources you can use, especially the length of a job you can run, depends on what kind of account you have.

What kind of account do I have?

  • If you work for an organization which is paying for access to Siku, you have a paid (pd-) account.
  • If you are in an academic research group which is not paying for access to Siku, you have a default (def-) account.
  • If you are an academic in a research group which has contributed equipment to Siku, you also have a contributed (ctb-) account in addition to a default account. Your jobs will normally be submitted under the contributed account, but you may change this by including the --account=def-??? directive in you job scripts.

If you want to find out what accounts are available to you, run sacctmgr show associations where user=$USER format=account%20 and examine the prefix on the account code:

  • pd-... is a paid account
  • ctb-... is a contributed account
  • def-... is a default account

What is the longest job I can run?

  • If you have a default account, you may use a time limit of up to 24 hours.
  • If you have a paid or contributed account, you may use a time limit of up to 7 days (168 hours).
  • Jobs requesting GPUs may not have a time limit longer than 24 hours, no matter what kind of account you have.

Jobs longer than 3 days are subject to some restrictions in order to keep waiting times from getting too long. See "Why hasn't my job started?" below for more about this.

There are a few nodes which accept only 3-hour jobs or shorter from most users (partition contrib3h). Consequently, jobs with a run time of 3 hours or less will often schedule much more quickly than other jobs. If you want an interactive job for testing purposes, we recommend salloc --time=3:0:0 (or less). If you can arrange your production workflow to include 3-hour jobs, those may benefit from more rapid scheduling.

How do I get a node with a GPU?

GPUs should be requested following these examples:

  • Request a Tesla V100 GPU:
#SBATCH --gres=gpu:v100:1
  • Request two RTX 6000 GPUs:
#SBATCH --gres=gpu:rtx6000:2

See "Node characteristics" below for the numbers and types of GPUs installed. Jobs requesting GPUs may request run times of up to 24 hours.

Why was my job rejected?

If you receive sbatch: error: Batch job submission failed: when you submit a job, the rest of the message will usually indicate the problem. Here are some you might see:

  • Invalid qos specification: You've used a --partition or --qos directive. Remove the directive and try again.
  • Requested time limit is invalid (missing or exceeds some limit)
    • def- accounts are limited to 24h.
    • GPU jobs are limited to 24h.
    • pd- and ctb- accounts are limited to 7d.
  • Invalid account or account/partition combination specified
    • Most Siku users do not need to supply --account; delete the directive and try again.
    • If you think you should have access to more than one account, see "What kind of account do I have?" above.

Why hasn't my job started?

The output from the sq and squeue commands includes a Reason field. Here are some Reasons you may see, and what to do about them.

  • Resources: The scheduler is waiting for other jobs to finish in order to have enough resources to run this job.
  • Priority: Higher priority jobs are being scheduled ahead of this one.
    • pd- and ctb- accounts always have higher priority than def- accounts. Within these tiers, priority is determined by the amount of computing that each account has done recently.
  • QOSGrpBillingMinutes: Paid account quarterly usage limit. See Tracking paid accounts.
  • QOSGrpCpuLimit: This may be due to:
    • The limit on the number of CPUs which can be used by long jobs at one time, considering all accounts in aggregate.
    • The limit on the number of CPUs which can be used by a ctb- account at one time. Jobs for a ctb- account run in the same priority tier as pd- jobs, but they cannot use more than a fixed amount of resources at any one time. Use --account=def-xxx to avoid this limit, but at the cost of priority
  • MaxCpuPerAccount: Long-job limit per account.
  • QOSMaxWallDurationPerJobLimit: Time limit >24h from def- account. This job will never run; scancel and resubmit with a time limit less than or equal to 24h.
  • ReqNodeNotAvail, Reserved for maintenance: There is maintenance scheduled to take place in the near future. Your job's time limit is long enough that it might not finish before the maintenance outage begins.
    • You can determine the start time of the maintenance outage with scontrol show reservations, and the current time with date.

Other restrictions

  • Each account may have no more than 5,000 jobs pending or running at one time. This is to prevent overloading the scheduler processes. A job array counts as multiple jobs.

Recent changes

  • 2022 February 08: Changes were introduced to permit paid and contributed accounts to run jobs of up to 7 days. No single account (research group) can have more than 600 CPUs of jobs longer than 3 days running at any one time, and no more than 800 CPUs in total can be running such long jobs. These numbers may be adjusted without notice. After 3 months we will review wait times and user satisfaction with this change.

Storage quotas and filesystem characteristics

Filesystem Default Quota Backed up? Purged? Mounted on Compute Nodes?
Home Space 52 GB and 512K files per user Yes No Yes
Scratch Space 20 TB and 1M files per user No Not yet implemented Yes
Project Space 1 TB and 512K files per group Yes No Yes

Node characteristics

Nodes Cores Available memory CPU Storage GPU
42 40 186G or 191000M 2 x Intel Xeon Gold 6248 @ 2.5GHz ~720G -
8 40 376G or 385024M 2 x Intel Xeon Gold 6248 @ 2.5GHz ~720G -
17 40 275G or 282000M 2 x Intel Xeon Gold 6248 @ 2.5GHz ~720G -
1 40 186G or 191000M 2 x Intel Xeon Gold 6148 @ 2.4GHz ~720G 3 x NVIDIA Tesla V100 (32GB memory)
1 40 186G or 191000M 2 x Intel Xeon Gold 6148 @ 2.4GHz ~720G 2 x NVIDIA Tesla V100 (32GB memory)
1 40 186G or 191000M 2 x Intel Xeon Gold 6248 @ 2.5GHz ~720G 4 x NVIDIA Quadro RTX 6000 (24GB memory)
1 40 186G or 191000M 2 x Intel Xeon Gold 6248 @ 2.5GHz ~720G 2 x NVIDIA Quadro RTX 6000 (24GB memory)
  • "Available memory" is the amount of memory configured for use by Slurm jobs. Actual memory is slightly larger to allow for operating system overhead.
  • "Storage" is node-local storage. Access it via the $SLURM_TMPDIR environment variable.
  • Hyperthreading is turned off.
  • All GPUs are PCIe-linked. NVLINK is not currently supported.

Operating system: CentOS 7

SSH host keys

ED25519 (256b)
SHA256:F9GcueU8cbB0PXnCG1hc4URmYYy/8JbnZTGo4xKflWU
MD5:44:2b:1d:40:31:60:1a:83:ae:1d:1a:20:eb:12:79:93
RSA (2048b)
SHA256:cpx0+k52NUJOf8ucEGP3QnycnVkUxYeqJQMp9KOIFrQ
MD5:eb:44:dc:42:70:32:f7:61:c5:db:3a:5c:39:04:0e:91