• Antelope Release 5.10 Linux CentOS release 7.6.1810 (Core) 3.10.0 2020-05-12

 

NAME

notes_mac_unix_anomalies - surprises that await Unix users new to Mac

DESCRIPTION

Apple makes powerful machines with a very nice user interface. However, the experienced Unix user is likely to have some painful lessons from time to time. This should provide some warning of some of these subtle issues.

Case Insensitive Filesystems

The default Mac filesystem -- what you will have unless you make a special effort -- is termed "case insensitive". This means the filesystem cannot distinguish between upper and lower case in the file name. For instance, the names INSTALL and the install refer to the very same file. The initial capitalization is indeed preserved from when the file is created, and that's what ls will show, but ls -l InStAlL will list a file with the letters install regardless of capitalization.

This is particularly piquant when you unpack a typical source tar package from the internet, and run the command make install -- often there's a file named INSTALL, and so make concludes that there is nothing to do, as the file install already exists.

To avoid this, whenever you get a new computer, run Disk Utility during the installation of Mac OS X, and recreate the disk partitions as Format Mac OS Extended (Case-sensitive) (with or without journaling). Of course, nothing is quite so simple. Apple claims to test all their programs with every supported filesystem, but other vendors are not so comprehensive. Programs from Adobe Systems in particular often break when installed on a case sensitive filesystem.

The launchd program

Apple has attempted to simplify the way in which processes can be started, so that persistent programs must be started through launchd. This leads to some results that can be surprising to those familiar with other Unix systems. If a program is started at a shell prompt, and put into the background, and then the user logs out, the background process loses access to directory services like getpwuid and dns. If the process tries to look up its own password entry, it will fail. If it tries to lookup www.google.com, or its own IP address, it will fail.

The workaround for now is to run such background tasks with /usr/libexec/StartupItemContext. This is just a script which runs the program with launchctl bsexec. See the Technical Note 2083 below for the lengthy and confusing official statement. Note that /usr/libexec/StartupItemContext has disappeared from OSX as of version 10.10 (Yosemite), requiring the replacement invocation of


/bin/launchctl bsexec /

Filesystem Fragmentation

Unix users accustomed to the UFS filesystem have not really had problems with fragmentation for the last twenty five years or so. But the HFS filesystem has significant problems with fragmentation, particularly if your applications like to write small bits of data to a variety of disk files over a long period of time. Apple's official advice in this situation is: Don't do that. Modern applications should instead read and write large amounts of data at once, not dribble stuff onto disk in small clumps. Presumably, that's part of the reason for the new logging application which Console taps into.

For a data collection center, which dribbles in data continuously from hundreds or thousands of sites, separating it out into many thousands of files, this admonition is not very useful. One workaround is to preallocate the files, (filling with null values, for instance), and then overwrite the values as the data becomes available. This leads to much, much, much better disk access when reading from these files, but does present a problem if all these files need to be created at a particular time (like a day boundary). An alternative is to write the files to a solid state disk, and later migrate them to spinning disk when they're complete. Just copying the files will essentially defragment them; Mac OS X has a built-in process which does this according to some internal rules, but it has not been adequate in some common situations for us.

The program hfsdebug from Amit Singh, the author of Mac OS X Internals, was potentially useful for investigating fragmentation issues. Evidently it's been replaced by the non-free program file Xray from http://filexray.com/ And hfsdebug won't work on releases after Snow Leopard.

From Dan Frakes, on a now-obsolete macintouch.com URL:

Matthew Popke wrote: "Mac OS X Panther does not really defragment your hard drive over the course of normal operation." This is incorrect. Panther does perform a certain degree of automatic defragmentation and file optimization. The former occurs when you open each file: OS X first checks to see if it's significantly fragmented. (The threshold is apparently 8 fragments.) If so, and if the file is under 20MB in size, the file is then defragmented by moving it to a contiguous block of space on your hard drive. This process won't affect files that are only slightly fragmented, or files larger than 20MB, but it should significantly reduce everyday file fragmentation. The latter works something like this: OS X keeps track how often files are read. Over periods of time it moves the most frequently read files ("hot files") to the sections of the hard drive platters that provide the best performance. (For example, sections of a drive platter closer to the center have better access times than sections near the outer edge.) This provides two benefits: Frequently read files are accessed more quickly and, in most cases, these files are defragmented in the process of being moved. Apparently this process only occurs for files 10MB or smaller in size. (For the curious, hot file data is stored in a file called .hotfile.btree at the root level of the boot volume.) Note that these features only function on journaled HFS Plus hard drives. (Journaling is enabled on the boot volume by default in Panther.) Because of these features, I generally recommend against using defragmentation and optimization utilities under Mac OS X unless you're regularly working with large data files (video, large databases, etc.) where fragmentation can significantly affect performance. And even then, I would recommend instead investing in a second hard drive to be used specifically for those files. Apple has a Knowledge Base article on defragmenting and optimizing drives at "About disk optimization with Mac OS X."

Hidden Files

The OS X Finder program will often hide some filesystem files it deems not appropriate to show to the user, especially in file-selection dialog boxes. There are a number of ways to get around this, documented on the web. A very convenient one is to invoke the following keyboard shortcut from within such a dialog box:

Shift-Command-.		[Shift-Command-Period]

This will toggle display of hidden folders such as /etc, /usr, and /opt.

The fprintf mutex

On Linux, multiple threads can write at the same time to a single file descriptor: the operating system (behind the scenes) puts a mutex in the output routine, to limit the interference between the multiple writes. Apple does not do this, and one consequence is that messages can just disappear.

SEE ALSO

notes_mac_bookshelf(5)
notes_mac_development(5)
notes_mac_extras(5)
notes_mac_setup(5)
notes_mac_mavericks(5)

AUTHOR

Daniel Quinlan
Printer icon