Showing posts with label filesystem. Show all posts
Showing posts with label filesystem. Show all posts

Thursday, March 20, 2014

Use fs_usage to monitor file system activity on OS X


Here's 3 commands I've found useful over the past year to see:

  • how much filesystem activity is going on
    IOPS (i/o operations per second), approximately
  • which applications are causing the most activity
  • which files or paths in the filesystem are being accessed
I will assume some unix familiarity here;  if you don't know where these commands go, then you probably don't understand the output either.  Come back after you've learned more.  All command lines are a single line, and appear below in courier bold.

Also, you need to be root for this.
I like to keep past outputs, and in a separate directory to contain clutter, so if you want to use these commands without modification then first do (one time only):
mkdir fs_usage

1.  Collect output from fs_usage

[42]original-mac 10:21:56 1082 ~# fs=$[fs+1]; fsd=0216; time fs_usage -w -f filesys  | grep '/.*/'  | head -10000 > fs_usage/fs.$fsd.$fs  


real    1m18.214s
user    0m1.379s
sys     0m1.428s


Here, only looking for the fs_usage output lines with pathnames (at least 2 slashes), the first 10k instances.  You can do more or less...  This took 78 seconds, which tells me the system is "fairly idle", at least mine, with the normal set of applications that I usually have open.  I have seen it complete in as little as two or three seconds.

I set $fsd as the date, and then increment the filename each time; this is just a personal packrat preference, but you can just use a single file in /tmp if you like.

In this form, it's easy to wrap with a while or
for sample in {1..10}; do ...; doneif you want to get ten samples, and then compare/combine the outputs.  Putting output from time into a separate file left as exercise for the reader.

Note these do not directly correspond to disk accesses (reads/writes), just general i/o activity.
'Activity Monitor' provides a graph of IOPS (or so it says); there are other benchmark programs if you want to check your disk (what's your favorite?).

2.  See which applications are issuing requests.  

Note with fs_usage '-w' wide output, the application name starts in column 234.  


[37]original-mac 10:02:40 1078 ~# cat fs_usage/fs.$fsd.$fs | cut -c234- | sort | uniq -c | sort -n | tail -25
  67 fseventsd.1560736
  67 fseventsd.3241
  67 fseventsd.639202
  80 CalendarAgent.2319030
  81 mds.2317581
  96 Dropbox.6042
 114 leapd.2991
 119 mdworker.2292251
 120 fseventsd.3427
 120 fseventsd.3428
 131 Google Chrome.1506982
 132 CalendarAgent.2319032
 148 Google Chrome.1507028
 152 launchd.210
 154 opendirectoryd.2318992
 154 opendirectoryd.2318994
 156 Google Chrome.1507026
 156 opendirectoryd.2318977
 160 opendirectoryd.2318972
 202 dbfseventsd.626141
 206 mtmfs.780
 230 Google Chrome.1506961
 288 com.apple.iCloud.2319050
 376 mtmd.781
6021 Google Chrome.1506955


Google Chrome is the majority of the activity...  not particularly surprising (I currently have 30+ tabs open).

3.  See what directories and files are being accessed

[36]original-mac 10:01:26 1077 ~# cat fs_usage/fs.$fsd.$fs | cut -c57-200 | sort | cut -c1-60 | uniq -c | sort -n | tail -33

  20 /Users/todd/Library/Application%2525252520Support
  20 /private/var/folders/98/vpw5gnnx1xg_9l3vjv3cc7r00000gn/T/.co
  20 0       /dev/disk0s2
  20 private/var/folders/98/vpw5gnnx1xg_9l3vjv3cc7r00000gn/T/.com
  21 /Applications/TextEdit.app
  22 /Users/todd/Library/Application%252520Support>>>>>>>>>>>>>>>
  23 /Users/todd/Library
  23 /Users/todd/Library/Application%20Support>>>>>>>>>>>>>>>>>>>
  23 /Users/todd/Library/Application%2520Support>>>>>>>>>>>>>>>>>
  24 /Users/todd/.dropbox/PENDING_MN7VKR
  24 /Users/todd/.dropbox/UPDATED_Xp1v55
  24 /Users/todd/Library/Application Support/Google/Chrome/.com.g
  24 /Users/todd/Library/Application Support/Google/Chrome/Local
  25 private/var/folders/98/vpw5gnnx1xg_9l3vjv3cc7r00000gn/T
  29 /Users/todd/Library/Saved Application State/com.apple.Termin
  31 /Users/todd
  31 /Users/todd/Library/Calendars
  32 00      /dev/disk0s3
  36 /Users/todd/Library/Application Support/iCloud>>>>>>>>>>>>>>
  40 /Users/todd/Library/Calendars/CE0DC09B-CC04-47D0-9217-E0E295
  42 /Users/todd/Library/Preferences/com.apple.security.revocatio
  47 /Volumes/rescue          
  58 /Users/todd/Library/Preferences/VMware Fusion/preferences
  72 /System/Library/CoreServices/backupd.bundle/Contents/Resourc
  89 /Users/todd/Library/Calendars/Calendar Cache-wal
 104 /Users/todd/Library/Calendars/Calendar Cache-journal
 114 /dev/console
 115 /Users/todd/Library/Application Support/Google/Chrome
 123 /private/var/folders/98/vpw5gnnx1xg_9l3vjv3cc7r00000gn/T
 216 /System/Library/PrivateFrameworks/CalendarAgent.framework/Ex
 250 /Users/todd/Library/Application Support/Google/Chrome/Safe B
 454 0       /dev/disk0s3
6487 /Users/todd/Library/Application Support/Google/Chrome/Defaul

Again, the fs_usage output has the pathname (since we explicitly grep'd for one) starting around column 57..  then I'm only looking at the first 60 characters of that path and aggregating to get a useful summary; you can use shorter or longer, or just search the raw output for specific files.

Mostly Chrome accessing my profile here.  Not sure about the /dev/disk accesses, or why some lines have '>>>>..' in them...  in the context of the full fs_usage output it might be more apparent.  However, for my purposes of approximately finding out what is going on, these are useful.

You need to run this a few times on yoru system to determine a baseline--i.e., what normal levels of activity are, and which applications are generating them.

DISKS ARE SLOW,  limited to one head per platter, and you have to wait for it to rotate into place.. a limiting resource (if you have an SSD or hybrid disk or RAID with battery-backed write-cache memory--they are a little better).  Sometimes you can't avoid them (e.g., you have to save your work).  But if/when your system starts slowing down, this can provide a little insight into what may be the bottleneck, and what unnecessary items you can shut down.


Tuesday, October 5, 2010

OSX: Constant disk activity after attaching a disk to Mac

Mac
(windows and linux, keep reading)

What it probably is: spotlight indexing.
If you're just hooking up a drive to copy some data or look for something, then this can be a hindrance to your work.

Verify that it is this process with:
sudo fs_usage

this will show all I/O activity, which will be a lot because it will include every line of text coming out to the terminal as well. Filter it with something like

sudo fs_usage | egrep -v 'screen|grep|head' | head -20

If you see mdworker, that's spotlight indexing going on.
If you see backupd, that's Time Machine (but you may have noticed the spinning icon in the menu bar).

Turn off indexing with:
mdutil -i off /Volumes/mountname

It may take 30 seconds or so (depending how busy it is) before you see
Indexing disabled.

You can turn it back on later by specifying 'on' instead of 'off'.

There are many other usages; you can use fs_usage to find what configuration or plist file(s) a program is using, where your data is going to on disk, etc.

Here's a MacWorld article: Watch file system activity in real time

Windows
On windows, get fsmon; it's one of the sysinternals tools.
regmon will do the same thing for registry accesses.

I've used fsmon to discover trojan processes before.
Windows == virus.

Linux
atop is supposed to come with ubuntu.
I don't think it gives specific paths in output, but once you have the pid you can connect with strace and just look at the IO syscalls, or use lsof to find which files are currently in use.



any other tools of note?


Enhanced by Zemanta

Friday, November 20, 2009

Mount filesystems over ssh, ftp, etc on OSX with MacFUSE, Macfusion, and a fix for snow leopard

OS X can already mount folders shared by windows machines with smbfs (Finder - Connect to server), or WebDAV natively. With a little more software, you can mount any (*nix) filesystem over ssh or ftp, with macFuse/Fusion.

'FUSE' is something like Filesystem in User SpacE.. that means it doesn't have to be part of the OS directly, most importantly making easy for developers to try something, test it, fix it, and try again really quick, instead of maybe having to compile a whole OS, rebooting, etc. This has been around for a long time on Linux, and a while for Macs too. Just a whole lot easier on a Mac because there's some nice interfaces.

  1. MacFUSE allows you to extend Mac OS X’s native file handling capabilities via 3rd-party file systems. Visit the MacFUSE website for more information.
  2. install, you will need to reboot.
  3. Get Macfusion from http://www.macfusionapp.org/ and install it.
  4. Run it and mount some drives. After it's mounted, you can exit Macfusion.
One problem, it didn't work for me.. because I have the newer 10.6 (snow leopard). The error I got (see the macfusion log viewer) was:

(macfusionAgent, SSHServerFS, norad, 11/20/09 7:35 PM) dyld: could not load inserted library: /Applications/Utilities/Macfusion.app/Contents/PlugIns/sshfs.mfplugin/Contents/Resources/sshnodelay.so

I put mine in Utilities, but you might also see the path
/Applications/Macfusion.app/Contents/PlugIns/sshfs.mfplugin/Contents/Resources/sshnodelay.so

tell it to look for beta versions, check for update, remove an old module, and then it works..


I see my password in the macfusion log viewer -- how to I get rid of that?