An understanding of file permissions is important to the success of computational jobs, and the security of your files.
The default settings are suitable for some, but not every use-case: without sufficient awareness, your files may be visible to people who should not be able to access them, and vice-versa.
We do not allow personally identifiable data to be stored on the cluster. If your data is personal please ensure it is stored correctly in line with the Data Protection policy. Please contact us if you are unsure, so we can discuss your options.
About half of all the storage support requests we get are regarding file permissions. This blog post will address common issues.
Every operating system uses file permissions, and while these may differ, they all follow the same principles.
The storage on Apocrita is based on the standard POSIX file permissions of User, Group, and Other. However, this is extended with Access Control Lists (ACLs) which make it possible to grant access to multiple groups or individuals in finer detail.
With standard POSIX/Linux permissions, each file has three sets of permissions, relating to owner, group and everyone else (commonly known as “other”). By default, new files are owned by your own username and default group.
The default group on the cluster is
qmul, although if you are an
external collaborator, it might be
Each file permission for the user, group and other can be assigned any of the following:
- read - user/group/other can read the file - signified by the ‘r’ character
- write - user/group/other can write to the file - signified by the ‘w’ character
- execute - user/group/other can run the file, or if it’s a directory they can list the contents - signified by the ‘x’ character
These will normally be listed as a series of letters but can also be represented in octal form with a digit for each of the user, group and “other” categories:
0 - no permissions 1 - ‘—x’ : execute 2 - ‘-w-’ : write 4 - ‘r—’ : read 3 : ‘-wx’ : writable and executable 5 : ‘r-x’ : readable and executable 6 : ‘rw-’ : readable and writable 7 : ‘rwx’ : read, write and execute.
You can see the permissions of a file by using
[abc123 ~]$ ls -l script.sh -rwxrw-r-- 1 abc123 qmul 1483838 Nov 9 2021 script.sh
rwxrw-r-- format means:
- owner has read, write and execute permission (
- group has read and write permission (
- everyone/other has read permission (
This can be represented in octal form as 761.
A common issue experienced by new users is receiving a permission denied error when trying to execute a script they have written. The error usually means that they need to add execute permissions to their file, rather than any other lack of user access privileges.
The default permissions are set based on the
umask, which specifies the
permissions which you don’t want to see by default on newly created files. It
can get a little complicated, since removal of a permission doesn’t necessarily
mean that all permission is granted when you don’t remove permissions. For
umask 077 ensures that
rwx is not applied to a new file. Note that
the user/owner execute permission is not added by the
touch command by
$ umask 077 $ mkdir dir $ touch file $ ls -ld dir file drwx------ 2 abc123 qmul 4096 2021-11-20 12:05 dir -rw------- 1 abc123 qmul 0 2021-11-20 12:05 file
Group shares have been designed to inherit group ownership from the parent directory - files created in a group share will be be limited to your group and not everyone.
A side-effect of this is that files created in your home directory
and later moved into a group share (with
mv) will keep the original
permissions and other members of the group will not have the correct
permissions. This issue is exacerbated if you create a file in a folder you
moved. If you are moving files into a group share it is always best to
copy them and then delete the originals for this reason.
You can temporarily change your current default group for the duration of your
login session with
newgrp <groupname>. This can be a time-saver when you’re
working with filesets owned by different groups, and using it before copying
and moving files around can help you ensure that file permissions are correct.
It’s not possible for standard users to change the ownership of a file. To claim ownership of a file owned by someone else, you can take a copy of a file - the resulting file will be owned by you. To delete the original file you will need someone who has write permissions on that file. You can always ask our team for help if you don’t have permissions to perform certain tasks. Where necessary, we will seek approval from the file owner first though.
You can change the permissions on any file you own. For example
chmod g-w file will revoke write access to
file by the group, and
g+r will add group read permission.
To add owner execute permissions to
$ chmod u+x file $ ls -l file -rwrx------ 1 abc123 qmul 0 2021-11-20 12:05 file
You can also change the group ownership of a file to any other group you are
also a member of, so if you’re a member of multiple groups you can do a
chgrp newgroup file which will change the group ownership of
file from the default
qmul group to
Access Control Lists
Access Control Lists extend the standard POSIX view of user/group/other and allow you to grant permission to additional groups and/or users. They are very flexible but can also cause more confusion.
While the standard Linux file ACL commands (such as
setfacl) may work, they can still give a limited view of what is really
going on. If you really want to see what the permissions are you should
mmsetacl. A detailed look at ACLs is out of scope for this
article, but it’s good be aware that if you have particular requirements that
aren’t satisfied by the standard permissions, we may be able to help.
Even More Permissions
The immutable flag which can be set with
mmchattr -i yes means a file can’t be
changed without unsetting this flag first. We use this on the
README file in
the top directory of a shared Research group area, so that the users within the
group can’t edit, change or remove that file.
Additionally, you may also come across sticky bit (commonly used on the
directory to allow everyone to write files that only the owner can delete) and
It’s important for all users to learn the basics of file permissions to ensure the visibility and ability to modify their data is neither too permissive, or too restrictive. Additionally, if cluster jobs fail due to permission issues, an understanding of the issues covered here will help you to troubleshoot why this occurred, and how to resolve the issue yourself, or escalating to us if administrative privileges are required.
Image Credit: Padlock by Maxim Zhgulev on Unsplash