Filesystems

Filesystem Hierarchy Standard

Kyle Rankin — Mon, 17 Jun 2019 11:00:00 +0000

What are these weird directories, and why are they there?

If you are new to the Linux command line, you may find yourself wondering why there are so many unusual directories, what they are there for, and why things are organized the way they are. In fact, if you aren't accustomed to how Linux organizes files, the directories can seem downright arbitrary with odd truncated names and, in many cases, redundant names. It turns out there's a method to this madness based on decades of UNIX convention, and in this article, I provide an introduction to the Linux directory structure.

Although each Linux distribution has its own quirks, the majority conform (for the most part) with the Filesystem Hierarchy Standard (FHS). The FHS project began in 1993, and the goal was to come to a consensus on how directories should be organized and which files should be stored where, so that distributions could have a single reference point from which to work. A lot of decisions about directory structure were based on traditional UNIX directory structures with a focus on servers and with an assumption that disk space was at a premium, so machines likely would have multiple hard drives.

/bin and /sbin

The /bin and /sbin directories are intended for storing binary executable files. Both directories store executables that are considered essential for booting the system (such as the mount command). The main difference between these directories is that the /sbin directory is intended for system binaries, or binaries that administrators will use to manage the system.

/boot

This directory stores all the bootloader files (these days, this is typically GRUB), kernel files and initrd files. It's often treated as a separate, small partition, so that the bootloader can read it more easily. With /boot on a separate partition, your root filesystem can use more sophisticated features that require kernel support whether that's an exotic filesystem, disk encryption or logical volume management.

/etc

The /etc directory is intended for storing system configuration files. If you need to configure a service on a Linux system, or change networking or other core settings, this is the first place to look. This is also a small and easy-to-back-up directory that contains most of the customizations you might make to your computer at the system level.

/home

The /home directory is the location on Linux systems where users are given directories for storing their own files. Each directory under /home is named after a particular user's user name and is owned by that user. On a server, these directories might store users' email, their SSH keys, or sometimes even local services users are running on high ports.

Go to Full Article

The Lustre Filesystem Dropped from the Linux 4.18 Kernel

Petros Koutoupis — Mon, 11 Jun 2018 14:52:52 +0000

by Petros Koutoupis

It's now official: the latest RC1 pull request for the Linux 4.18 will not host the nearly 15-year-old Lustre filesystem.

Greg Kroah-Hartman has been growing weary of the team developing its source code not pushing cleaner and fixed code to the staging tree. The removal was committed on June 5, 2018: with the following notes:

The Lustre filesystem has been in the kernel tree for over 5 years now. While it has been an endless source of enjoyment for new kernel developers learning how to do basic coding style cleanups, as well as a semi-entertaining source of bewilderment from the vfs developers any time they have looked into the codebase to try to figure out how to port their latest api changes to this filesystem, it has not really moved forward into the "this is in shape to get out of staging" despite many half-completed attempts.

And getting code out of staging is the main goal of that portion of the kernel tree. Code should not stagnate, and it feels like having this code in staging is only causing the development cycle of the filesystem to take longer than it should. There is a whole separate out-of-tree copy of this codebase where the developers work on it, and then random changes are thrown over the wall at staging at some later point in time. This dual-tree development model has never worked, and the state of this codebase is proof of that.

So, let's just delete the whole mess. Now the lustre developers can go off and work in their out-of-tree codebase and not have to worry about providing valid changelog entries and breaking their patches up into logical pieces. They can take the time they have spent doing those types of housekeeping chores and get the codebase into a much better shape, and it can be submitted for inclusion into the real part of the kernel tree when ready.

Honestly, I do not blame him. The staging tree is primarily intended for unstable and less than mature code, which ideally should move to the mainline within a short time of further development. It's a temporary (that is, staging) location. It's not that I don't appreciate the Lustre filesystem. In fact, I once wrote about it for Linux Journal in the past.

For those who are less familiar with this filesystem: Lustre (or Linux Cluster) is a distributed filesystem typically deployed in large-scale cluster computing environments. Lustre is designed to be both performant and to scale to tens of thousands of nodes and to petabytes of storage. And as what may have just been alluded to already, a distributed filesystem allows access to files from multiple hosts sharing a computer network.

Go to Full Article

ZFS for Linux

Charles Fisher — Mon, 12 Feb 2018 15:54:56 +0000

by Charles Fisher

Presenting the Solaris ZFS filesystem, as implemented in Linux FUSE, native kernel modules and the Antergos Linux installer.

ZFS remains one of the most technically advanced and feature-complete filesystems since it appeared in October 2005. Code for Sun's original Zettabyte File System was released under the CDDL open-source license, and it has since become a standard component of FreeBSD and slowly migrated to various BSD brethren, while maintaining a strong hold over the descendants of OpenSolaris, including OpenIndiana and SmartOS.

Oracle is the owner and custodian of ZFS, and it's in a peculiar position with respect to Linux filesystems. Btrfs, the main challenger to ZFS, began development at Oracle, where it is a core component of Oracle Linux, despite stability issues Red Hat's recent decision to deprecate Btrfs likely introduces compatibility and support challenges for Oracle's Linux road map. Oracle obviously has deep familiarity with the Linux filesystem landscape, having recently released "dedup" patches for XFS. ZFS is the only filesystem option that is stable, protects your data, is proven to survive in most hostile environments and has a lengthy usage history with well understood strengths and weaknesses.

ZFS has been (mostly) kept out of Linux due to CDDL incompatibility with Linux's GPL license. It is the clear hope of the Linux community that Oracle will re-license ZFS in a form that can be included in Linux, and we should all gently cajole Oracle to do so. Obviously, a re-license of ZFS will have a clear impact on Btrfs and the rest of Linux, and we should work to understand Oracle's position as the holder of these tools. However, Oracle continues to gift large software projects for independent leadership. Incomplete examples of Oracle's largesse include OpenOffice and recently Java Enterprise Edition, so it is not inconceivable that Oracle's generosity may at some point extend additionally to ZFS.

Go to Full Article

Paragon Software Group's Paragon ExtFS for Mac

James Gray — Mon, 18 Sep 2017 11:11:10 +0000

by James Gray

Ever more Mac aficionados are discovering the virtues of Linux, especially when their older hardware can experience a renaissance. One annoying barrier to dual-boot nirvana is filesystem incompatibility, whereby the Linux side can access the Mac side, but Apple's macOS doesn't support Linux drives at all—not even in read-only mode.

A handy solution is Paragon Software Group's Paragon ExtFS for Mac, a low-level macOS filesystem driver designed to eliminate filesystem incompatibility between Linux and Mac operating systems. The solution grants transparent read/write access to ExtFS Linux partitions on macOS, allowing Mac users to access files fully that are stored on Linux volumes hassle-free, streamline data sharing and transfer up to 4GB+ files at a high rate.

The new 11th edition of ExtFS for Mac features a completely new UI and advanced features, including extended mounting options and a menu bar app allowing users one-click access to all ExtFS drives and instant execution of the most common volume operations—for example, volume mount, unmount and verify. Large volume support enables mounting volumes more than 2TB in size. ExtFS for Windows also is available from the company.

Go to Full Article

Paragon Software Group's ExtFS for Windows

James Gray — Mon, 07 Nov 2016 13:50:04 +0000

by James Gray

Fellow Linux/Windows dual-booters out there are familiar with this problem: you can access Windows files from your Linux session, but not the other way around. Saving humanity by chipping away at Windows' illogic is ExtFS for Windows, a handy utility from the Paragon Software Group that gives dual-boot users full read-write access to Linux partitions from their Windows session.

ExtFS' drivers are based on Paragon's proprietary cross-platform Universal File System Driver (UFSD) technology, which provides a higher data transfer rate than native filesystem performance. Paragon recently unveiled a new feature-rich version 4 of ExtFS for Windows, which comes with extended support for Ext4 file formats enabling the highest transfer speeds and mounts Linux volumes, including those more than 2TB in size, at up to twice the rate of the previous release.

Thanks to the development of ExtFS, Paragon calls itself the first software developer to implement a full set of drivers with complete read and write access to partitions on all popular filesystems. ExtFS for Mac is also available from the company.

Go to Full Article

ZFS: Finding Its Way to a Linux Near You?

Petros Koutoupis — Wed, 01 Jun 2016 18:00:00 +0000

by Petros Koutoupis

It seems like only yesterday that I read Jeff Bonwick's blog entry "ZFS: The Last Word in Filesystems". It was Halloween of 2005 that ZFS was fully integrated into Sun Microsystem's Solaris, and the filesystem was very well received. For the readers not familiar with ZFS, it is a combined all-purpose filesystem and volume manager. It simplified data storage management while also offering the most advanced features of the time. Such technologies include drive pooling with software RAID support, file snapshots, in-line data compression, data deduplication, built-in data integrity, advanced caching (to DRAM and SSD), and more. Today, the ZFS trademark and technology is owned and maintained by the Oracle Corporation.

Also in 2005, Sun Microsystems introduced OpenSolaris. Now a defunct project, OpenSolaris was a fully functional Solaris operating system built entirely from open source, which included ZFS, and all of which were re-licensed to the Common Development and Distribution License (CDDL), a weak copyleft license based on the Mozilla Public License (MPL). Although open source, ZFS and anything else under the CDDL was, and supposedly still is, incompatible with the GNU General Public License (GPL). This includes the Linux kernel and eventually would lead to the birth of Btrfs.

To avoid licensing infringements, the earliest incarnations of ZFS on Linux were written for the Filesystem in Userspace (FUSE). This prevented the technology from touching the Linux kernel. It also added its fair share of limitations. Being in userspace, it never could really measure up to its Solaris and FreeBSD counterparts. Over time, some of the FUSE implementations were highly neglected and in some cases abandoned. In 2008, the "ZFS on Linux" project changed everything by developing an in-kernel implementation of ZFS. Since its conception, the project was met with a lot of resistance (and criticism) from within the Linux community, all relating to licensing.

Fast-forward to the present, and two distributions have challenged this. Last month, Canonical, the parent company of the Ubuntu Linux distribution released the latest Ubuntu 16.04, codenamed Xenial Xerus. One of the most noteworthy additions to this release was the full integration of pre-built ZFS modules. Although Canonical now ships Ubuntu with ZFS, it has publicly stated that its legal team did not see a violation of the GPL. This matter is still being debated.

Shortly following this news and through a separate and completely unrelated effort, the Debian distribution announced the inclusion of the ZFS source code, buildable via the Dynamic Kernel Module Support (DKMS) framework. However, it is not provided in the "main" section archive but instead in "contrib". Under the legal advice of the Software Freedom Law Center, this approach is seen as not violating the GPL license.

Go to Full Article