In my other thread, someone posted regarding defragging for Panther. This was posted:

http://nslog.com/archives/2003/10/26/qotd_fragmentation.php

...which cites this:

http://article.gmane.org/gmane.comp.macosx.general/22906

quote:

---

While we're talking about the not so well known/hidden new Panther features: while digging through the kernel sources i've found a quite amusing new feature built into HFS+. Namely automatic file defragmentation.

Everytime an application opens a file for reading, HFS+ checks if the file is fragmented and is less than 20MB in size. If so, it copies the file's contents to a continuous region on the disk and frees up the previously allocated blocks.

---

What's the story on this? Why wasn't it mentioned earlier?

God knows why it wasn't mentioned before, but if it's true, that's awesome. Now I don't even need to erase and install.

[This message was edited by Caesar on October 30, 2003 at 18:56.]

I've got plenty of files that are fragmented and over 20MB, though.

(And Drive 10 said my disk was very fragmented after heavy use under Panther 7B68, as well)

cool. i'd like some more confirmation, but, cool.

This is neat. Overall, it will reduce file fragmentation making some speed increases on access, but won't reduce filesystem fragmentation (per file versus the entireity of the disc), which is why Drive10 will still say that your disc is heavily fragmented.

At least, that's what my take on it would be.

quote:

---

Originally posted by Mr VacBob:
(And Drive 10 said my disk was very fragmented after heavy use under Panther 7B68, as well)

---

If this story is true, it seems it would on defrag on a file-by-file (as needed) basis.

Perhaps your volume's heavy fragmentation was from files that were not yet accessed under Panther?

If this is the case, the fragmented files shouldn't matter; if they ain't being accessed, leave 'em alone!

quote:

---

Originally posted by [Fuzzy]:
This is neat. Overall, it will reduce file fragmentation making some speed increases on access, but won't reduce filesystem fragmentation (per file versus the entireity of the disc), which is why Drive10 will still say that your disc is heavily fragmented.

---

Doesn't OS X try to cache most of the filesystem in RAM anyway?

As I posted in hanser's iBook G4 thread, and as fuzzy notes:

I would like to see some official Apple confirmation of this. Also, this doesn't help to optimize the placement of files on the HD. You still don't want your OS, App executables, and free space (read VM) on the slowest part of the drive. At the moment, only Norton Utilities optimizes that.


Does Drive 10 optimize file placement on HDs and optimize the btree catalogue? I was under the impression that it only de-fragmented files (but did not optimize file location).

The feature sounds neat, but I'd actually prefer not to have this going on all the time. I'd turn it off.

I'd be perfectly satisfied if Apple simply added a basic, reliable defrag utility and let me run it whenever necessary.

quote:

---

I would like to see some official Apple confirmation of this.

---

How's this?

static int
hfs_open(ap)
	struct vop_open_args /* {
		struct vnode *a_vp;
		int  a_mode;
		struct ucred *a_cred;
		struct proc *a_p;
	} */ *ap;
{
	struct vnode *vp = ap->a_vp;
	struct filefork *fp = VTOF(vp);
	struct timeval tv;

	/*
	 * Files marked append-only must be opened for appending.
	 */
	if ((vp->v_type != VDIR) && (VTOC(vp)->c_flags & APPEND) &&
	    (ap->a_mode & (FWRITE | O_APPEND)) == FWRITE)
		return (EPERM);

	if (ap->a_mode & O_EVTONLY) {
		if (vp->v_type == VREG) {
			++VTOF(vp)->ff_evtonly_refs;
		} else {
			++VTOC(vp)->c_evtonly_refs;
		};
	};

	/*
	 * On the first (non-busy) open of a fragmented
	 * file attempt to de-frag it (if its less than 20MB).
	 */
	if ((VTOHFS(vp)->hfs_flags & HFS_READ_ONLY) ||
	    !UBCISVALID(vp) || ubc_isinuse(vp, 1)) {
		return (0);
	}
	fp = VTOF(vp);
	if (fp->ff_blocks &&
	    fp->ff_extents[7].blockCount != 0 &&
	    fp->ff_size <= (20 * 1024 * 1024)) {
		/* 
		 * Wait until system bootup is done (3 min).
		 */
		microuptime(&tv;);
		if (tv.tv_sec < (60 * 3)) {
			return (0);
		}
		(void) hfs_relocate(vp, VTOVCB(vp)->nextAllocation + 4096, ap->a_cred, ap->a_p);
	}

	return (0);
}

/*
 * Relocate a file to a new location on disk
 *  cnode must be locked on entry
 *
 * Relocation occurs by cloning the file's data from its
 * current set of blocks to a new set of blocks. During
 * the relocation all of the blocks (old and new) are
 * owned by the file.
 *
 * -----------------
 * |///////////////|
 * -----------------
 * 0               N (file offset)
 *
 * -----------------     -----------------
 * |///////////////|     |               |     STEP 1 (aquire new blocks)
 * -----------------     -----------------
 * 0               N     N+1             2N
 *
 * -----------------     -----------------
 * |///////////////|     |///////////////|     STEP 2 (clone data)
 * -----------------     -----------------
 * 0               N     N+1             2N
 *
 *                       -----------------
 *                       |///////////////|     STEP 3 (head truncate blocks)
 *                       -----------------
 *                       0               N
 *
 * During steps 2 and 3 page-outs to file offsets less
 * than or equal to N are suspended.
 *
 * During step 3 page-ins to the file get supended.
 */
__private_extern__
int
hfs_relocate(vp, blockHint, cred, p)
	struct  vnode *vp;
	u_int32_t  blockHint;
	struct  ucred *cred;
	struct  proc *p;
{
	struct  filefork *fp;
	struct  hfsmount *hfsmp;
	ExtendedVCB *vcb;

	u_int32_t  headblks;
	u_int32_t  datablks;
	u_int32_t  blksize;
	u_int32_t  realsize;
	u_int32_t  growsize;
	u_int32_t  nextallocsave;
	u_int32_t  sector_a;
	u_int32_t  sector_b;
	int eflags;
	u_int32_t  oldstart;  /* debug only */
	off_t  newbytes;
	int  retval;

	if (vp->v_type != VREG && vp->v_type != VLNK) {
		return (EPERM);
	}
	
	hfsmp = VTOHFS(vp);
	if (hfsmp->hfs_flags & HFS_FRAGMENTED_FREESPACE) {
		return (ENOSPC);
	}

	fp = VTOF(vp);
	if (fp->ff_unallocblocks)
		return (EINVAL);
	vcb = VTOVCB(vp);
	blksize = vcb->blockSize;
	if (blockHint == 0)
		blockHint = vcb->nextAllocation;

	if ((fp->ff_size > (u_int64_t)0x7fffffff) ||
	    (vp->v_type == VLNK && fp->ff_size > blksize)) {
		return (EFBIG);
	}

	headblks = fp->ff_blocks;
	datablks = howmany(fp->ff_size, blksize);
	growsize = datablks * blksize;
	realsize = fp->ff_size;
	eflags = kEFContigMask | kEFAllMask | kEFNoClumpMask;
	if (blockHint >= hfsmp->hfs_metazone_start &&
	    blockHint <= hfsmp->hfs_metazone_end)
		eflags |= kEFMetadataMask;

	hfs_global_shared_lock_acquire(hfsmp);
	if (hfsmp->jnl) {
		if (journal_start_transaction(hfsmp->jnl) != 0) {
			return (EINVAL);
		}
	}

	/* Lock extents b-tree (also protects volume bitmap) */
	retval = hfs_metafilelocking(hfsmp, kHFSExtentsFileID, LK_EXCLUSIVE, p);
	if (retval)
		goto out2;

	retval = MapFileBlockC(vcb, (FCB *)fp, 1, growsize - 1, §or;_a, NULL);
	if (retval) {
		retval = MacToVFSError(retval);
		goto out;
	}

	/*
	 * STEP 1 - aquire new allocation blocks.
	 */
	nextallocsave = vcb->nextAllocation;
	retval = ExtendFileC(vcb, (FCB*)fp, growsize, blockHint, eflags, &newbytes;);
	if (eflags & kEFMetadataMask)                   
		vcb->nextAllocation = nextallocsave;

	retval = MacToVFSError(retval);
	if (retval == 0) {
		VTOC(vp)->c_flag |= C_MODIFIED;
		if (newbytes < growsize) {
			retval = ENOSPC;
			goto restore;
		} else if (fp->ff_blocks < (headblks + datablks)) {
			printf("hfs_relocate: allocation failed");
			retval = ENOSPC;
			goto restore;
		}

		retval = MapFileBlockC(vcb, (FCB *)fp, 1, growsize, §or;_b, NULL);
		if (retval) {
			retval = MacToVFSError(retval);
		} else if ((sector_a + 1) == sector_b) {
			retval = ENOSPC;
			goto restore;
		} else if ((eflags & kEFMetadataMask) &&
		           ((((u_int64_t)sector_b * hfsmp->hfs_phys_block_size) / blksize) >
		              hfsmp->hfs_metazone_end)) {
			printf("hfs_relocate: didn't move into metadata zone\n");
			retval = ENOSPC;
			goto restore;
		}
	}
	if (retval) {
		/*
		 * Check to see if failure is due to excessive fragmentation.
		 */
		if (retval == ENOSPC &&
		    hfs_freeblks(hfsmp, 0) > (datablks * 2)) {
			hfsmp->hfs_flags |= HFS_FRAGMENTED_FREESPACE;
		}
		goto out;
	}

	fp->ff_size = fp->ff_blocks * blksize;
	if (UBCISVALID(vp))
		(void) ubc_setsize(vp, fp->ff_size);

	/*
	 * STEP 2 - clone data into the new allocation blocks.
	 */

	if (vp->v_type == VLNK)
		retval = hfs_clonelink(vp, blksize, cred, p);
	else if (vp->v_flag & VSYSTEM)
		retval = hfs_clonesysfile(vp, headblks, datablks, blksize, cred, p);
	else
		retval = hfs_clonefile(vp, headblks, datablks, blksize, cred, p);

	if (retval)
		goto restore;
	
	oldstart = fp->ff_extents[0].startBlock;

	/*
	 * STEP 3 - switch to clone and remove old blocks.
	 */
	SET(VTOC(vp)->c_flag, C_NOBLKMAP);   /* suspend page-ins */

	retval = HeadTruncateFile(vcb, (FCB*)fp, headblks);

	CLR(VTOC(vp)->c_flag, C_NOBLKMAP);   /* resume page-ins */
	if (ISSET(VTOC(vp)->c_flag, C_WBLKMAP))
		wakeup(VTOC(vp));
	if (retval)
		goto restore;

	fp->ff_size = realsize;
	if (UBCISVALID(vp)) {
		(void) ubc_setsize(vp, realsize);
		(void) vinvalbuf(vp, V_SAVE, cred, p, 0, 0);
	}

	CLR(VTOC(vp)->c_flag, C_RELOCATING);  /* Resume page-outs for this file. */
out:
	(void) hfs_metafilelocking(VTOHFS(vp), kHFSExtentsFileID, LK_RELEASE, p);

	retval = VOP_FSYNC(vp, cred, MNT_WAIT, p);
out2:
	if (hfsmp->jnl) {
		if (VTOC(vp)->c_cnid < kHFSFirstUserCatalogNodeID)
			(void) hfs_flushvolumeheader(hfsmp, MNT_WAIT, HFS_ALTFLUSH);
		else
			(void) hfs_flushvolumeheader(hfsmp, MNT_NOWAIT, 0);
		journal_end_transaction(hfsmp->jnl);
	}
	hfs_global_shared_lock_release(hfsmp);

	return (retval);

restore:
	/*
	 * Give back any newly allocated space.
	 */
	if (fp->ff_size != realsize)
		fp->ff_size = realsize;
	(void) TruncateFileC(vcb, (FCB*)fp, fp->ff_size, false);
	if (UBCISVALID(vp))
		(void) ubc_setsize(vp, fp->ff_size);
	CLR(VTOC(vp)->c_flag, C_RELOCATING);
	goto out;
}

quote:

---

Originally posted by King Friday:
The feature sounds neat, but I'd actually prefer not to have this going on all the time. I'd turn it off.

---

Why?

quote:

---

I'd be perfectly satisfied if Apple simply added a basic, reliable defrag utility and let me run it whenever necessary.

---

Well, that's a much, much, harder 'app' to write than what they've done here, since the 'app' has to keep a map of files, fragmented, files, and spaces, and somehow create an optimized 'map' containing as many unfragmented files and as few open spaces as possible, all while doing it as quickly as possible...

Which isn't easy ^^

can someone translate what John said into English?

quote:

---

Originally posted by BEIGE:
can someone translate what John said into English?

---

Yeah, I think he said "Here is your confirmation TigerKR. Please have a good day now." I know the language but the dialect is foreign to me.

/*
 * Relocate a file to a new location on disk
 *  cnode must be locked on entry
 *
 * Relocation occurs by cloning the file's data from its
 * current set of blocks to a new set of blocks. During
 * the relocation all of the blocks (old and new) are
 * owned by the file.
 *
 * -----------------
 * |///////////////|
 * -----------------
 * 0               N (file offset)
 *
 * -----------------     `´`´`´`´`´`´`´`´`
 * |///////////////|     }    whirr...   {     STEP 1 (aquire new blocks)
 * -----------------     `´`´`´`´`´`´`´`´`
 * 0               N     N+1             2N
 *
 * -----------------     -----------------
 * |       ////////| ===}|///////        |     STEP 2 (clone data)
 * -----------------     -----------------
 * 0               N shhhwip!            2N
 *
 *                       -----------------
 *     :>POOF!<:         |////*gleam*////|     STEP 3 (head truncate blocks)
 *                       -----------------
 *                       0               N
 *
 * During steps 2 and 3 page-outs to file offsets less
 * than or equal to N are suspended.
 *
 * During step 3 page-ins to the file get supended.
 */

Leave it to Apple to draw diagrams in their comment blocks. (Okay, I may have embellished a bit for Aqua effect.)

And I'll take that file-block-moving diagram as confirmation -- if John will tell us where he found it...

FYI. Its called "Hot-File-Adaptive-Clustering". :-)

quote:

---

Originally posted by iconmaster:
And I'll take that file-block-moving diagram as confirmation -- if John will tell us where he found it...

---

Presumably from darwin kernel source..

quote:

---

Originally posted by xrules:
FYI. Its called "Hot-File-Adaptive-Clustering". :-)

---

Hot File-on-File action!

I wanna program with sound effects. We know some of the programmers are having fun.

I could see this being the root of the file errors on External Firewire Hard Drives. If there was a bug in this code, the cloning data to a new region would result in a bunch of gibberish and data loss.

quote:

---

I'll take that file-block-moving diagram as confirmation -- if John will tell us where he found it...

---

That was from the Darwin 7.0 source code.

TigerKR,

Does the latest Norton Utilities (specifically Disk Doctor & Speed Disk) work with Panther? I'm talking about the Boot CD.

I'm too scared to try it out.
RW

uh, I gotta ask, does it really say 'shhhwip' and Poof! in the comments?

If so I need to really review my coding style guide! Clearly I'm not having enough fun..

quote:

---

Originally posted by stephenb:
I wanna program with sound effects. We know some of the programmers are having fun.

---

quote:

---

Originally posted by storme:
uh, I gotta ask, does it really say 'shhhwip' and Poof! in the comments?

---

Sorry guys -- "embellished a bit" as it says above. Let's not start any rumors about the Apple developers turning their code into ASCII comic strips; we'd never hear the end of it in the Battlefront.

I'm very happy to see this.

Question- how does this bode for more extensive chron jobs run during the off hours? Shouldn't I be able to schedule a more extensive defragging at 4 am?


Second question- What count's as a fragmented file? I don't really worry about a file that's in two or three pieces. It's the system files that end up in 30 or 40 pieces that are a real problem, and I'm not really sure I want the OS defragging that file when I'm opening it ('cause that's when I need it).

I'd much rather have the OS keeping track of which files are fragmented (maybe flagging the worst offenders/prioritizing) and then defragging them when the system is idle.

Don't get me wrong, I'm very happy to see this autodefrag implementation, I'm just a little worried how it might impact performance, especially on a full HD. I suppose any performance hit will largely be compensated for by performance gains once this system's run for a while, hmm?

[This message was edited by Ashby on October 29, 2003 at 11:57.]

I guess you could just write a script that opens a bunch of your documents, then closes everything down a few minutes later. Maybe on Monday nights, everything in /Documents gets opened (and defragged by the system), /Music on Tuesday, /Pictures on Wednesday, etc..

This must be true, as I booted up on my Norton 6 disk and using SpeedDisk I have only 100 fragmented files out of 150,000. This is also after multiple Panther beta installs too.

quote:

---

Originally posted by Rollins:
Does the latest Norton Utilities (specifically Disk Doctor & Speed Disk) work with Panther? I'm talking about the Boot CD.

---

To be honest, my family pack is non-opened (and of course non-installed) at the moment because I'm getting a new business off of the ground. However, Symantec has this to say:

quote:

---

Symantec 2003-2004 product compatibility with Mac OS X 10.3 (code named Panther)

Situation:
This document outlines compatibility between Symantec 2003-2004 products and Mac OS X 10.3 (code named Panther).

Solution:
Symantec Macintosh software is not yet fully compatible with Apple's new operating system Mac OS X 10.3 (code named Panther), released October 24, 2003. Symantec is currently working on updates for compatibility with Mac OS X 10.3. This document will be updated when additional information is available. Bookmark this page and check back for news.

If you are running Mac OS X 10.3 and a Symantec 2003-2004 Macintosh product, refer to the section for your product below for compatibility information.

...

Norton SystemWorks for Macintosh 3.0.x

In Mac OS X 10.3 (code named Panther), no disks appear in the Norton Disk Doctor, UnErase, Volume Recover or Speed Disk window. Norton FileSaver runs in the background at startup, but no scheduled FileSaver events run.

To use Norton Utilities on a Macintosh with Mac OS X 10.3 installed, you must start up from the Norton Utilities or Norton SystemWorks CD, a second partition, or an external hard drive with Mac OS X 10.1.5 -10.2.8 and Norton Utilities installed. Another solution, if you have a second Macintosh with FireWire, is to use FireWire target disk mode to examine the hard disk on your Macintosh with Mac OS X 10.3 installed. Read the document How to use FireWire target disk mode to run Norton Utilities from a second computer.

Norton AntiVirus Auto-Protect works in Mac OS X 10.3 to protect your computer from viruses. You may also scan disks with Norton AntiVirus for Macintosh in Mac OS X 10.3. When "fast user switching" is enabled, Norton AntiVirus alerts appear only for the user that first logged in. If another user is active when the alert occurs, the alert does not appear for that user.

You must run LiveUpdate manually in Mac OS X 10.3. Norton Scheduler does not work in Mac OS X 10.3, so scheduled events for Norton AntiVirus, Norton FileSaver, Speed Disk or LiveUpdate do not run.

Symantec is currently working on updates for compatibility with Mac OS X 10.3 and this document will be updated when one is available.

---

Long story short, you have to boot from the Norton CD or another non-Panther partition/drive in order to use Norton (which you would do for repairs and optimization anyway), and in that scenario, it will work.

quote:

---

Originally posted by iconmaster:
I guess you could just write a script that opens a bunch of your documents, then closes everything down a few minutes later. Maybe on Monday nights, everything in /Documents gets opened (and defragged by the system), /Music on Tuesday, /Pictures on Wednesday, etc..

---

Very creative idea, I wouldn't have thought of that - unfortunately you'd want to restart after that, because your system would likely start paging like a mofo. Also, opening documents and saving them could introduce problems with save and replace dialogue boxes and possible application errors.

quote:

---

What count's as a fragmented file?

---

As far as the auto-defrag code in conserned, this:

if (fp->ff_blocks &&
    fp->ff_extents[7].blockCount != 0 &&
    fp->ff_size <= (20 * 1024 * 1024))

which roughly means "files less than 20MB that have a non-zero block count in the last item of their list of extents structures." You'll have to chase all those structs through the code to find out if that necessarily indicates fragmentation or is just a best-guess.

quote:

---

Originally posted by Rollins:
TigerKR,

Does the latest Norton Utilities (specifically Disk Doctor & Speed Disk) work with Panther? I'm talking about the Boot CD.

I'm too scared to try it out.
RW

---

I was brave and used Speed Disk v6 from the Norton Startup CD which had 9.04 to optimize my Panther install. I have never had problems with SD, but I won't touch Disk Doctor any longer.

quote:

---

Originally posted by TigerKR:
Very creative idea, I wouldn't have thought of that

---

Thanks. IANAP, but I like problem-solving.

quote:

---

opening documents and saving them could introduce problems with save and replace dialogue boxes and possible application errors.

---

Doesn't the defragmentation happen on opening the file? As I understand it, saving wouldn't be necessary -- just a lot of opening and closing.

quote:

---

unfortunately you'd want to restart after that, because your system would likely start paging

---

Agreed. Perhaps the script could end with a restart command; or you could turn on Panther's scheduled shutdown feature, timed sufficiently later than the script.

quote:

---

Originally posted by iconmaster:
...

quote:

---

unfortunately you'd want to restart after that, because your system would likely start paging

---

Agreed. Perhaps the script could end with a restart command; or you could turn on Panther's scheduled shutdown feature, timed sufficiently later than the script.

---

Why is this necessary? So, it starts paging in and out in the middle of the night. It'll be done by morning, and the memory will be freed in due time. My concern is not so much about uptime () but about having to close stuff down and then starting it back up after login.

-Ster

quote:

---

Originally posted by The Limey:
God knows why it wasn't mentioned before, but if it's true, that's awesome. Now I don't even need to erase and install.

---

Word to that my friend.

and Mr VacBob: I erased and isntalled panther on the iBook I sold, and Drive 10 v1.1.4 said it was mucho fragmented.

Funny that.

That could be the most beautiful and best documented code I have ever seen.

Has anyone verified that this is enabled by default?

quote:

---

Has anyone verified that this is enabled by default?

---

I don't see how it could be disabled.

John,

Is that code executed when the file is actually read, or when the file is just opened? I was thinking that it would be fairly straightforward to `find / -type f` to list all the files, and pipe that to a program (or script?) that would just call `open()` and then `close()` it back again. If it defrags on `open()`, then that would end up defragging all the eligible files, w/o the overhead of having to launch the proper program or deal with saving it and all that stuff.

-Ster

As far as I can tell from reading the code:

HFS+ references files using "extents". An extent is a data structure in the HFS catalog file (the file that keeps track of all the other files in the system). A given extent stores a start location, and a length - essentially, it tracks a single contiguous chunk of bytes on disk. As a file becomes fragmented, it uses more and more extents to keep track of it (i.e. a 100 byte file might be tracked in a 1 extent of 20 bytes, 3 extents of 10 bytes each, and 1 extent of 50 bytes). Up to 8 extents can be tracked directly in the catalog tree. If a file has more than 8 extents, they are tracked in another file - the Extents Overflow File. Access to a file that is tracked in both the Catalog and the Extents Overflow File is slow - both because it implies that the file is (relatively) highly fragmented, and because the Extents Overflow File probably isn't as "hot" as the Catalog file (not as likely to be cached). In general, access to a fragmented file is slow.

What this code is doing is: When a file is opened, OSX will look at it. If the file has any data, if the file is less than 20MB in size, and if the file has 8 or more extents, it will be defragmented. Note that the defragmentation won't happen if the disk has very little contiguous freespace left, since you don't have the space to defragment.

I'd guess that the parameters (20MB and 8+ extents) were gathered through some sort of research or testing. It seems high for my taste, but I'll trust the engineers (particularly since Panther seems pretty fast . My guess as to why this happens _all_ the time: Willingness to pay an up front cost for speed in subsequent operations. There might also be the possibility that most of the time an open is followed by a read, so defragging might not do much more than preheat the file-cache. I'm not sure what "non-busy" means in the code, I'll have to do more digging. One nice thing about this scheme is that you don't spend time moving around files that no-one cares about. If you don't use it, it doesn't get defragged.

The rest of the hotfile mechanism is interesting, and hasn't been discussed yet - it looks like over a period of time, OSX keeps track of which files have been use the most and moves them to the "hot band". This is already a bonus, since it'll defrag the files during the move, but I'm not sure what the "hot band" is (probably the fastest area on the physical hard disk, but I'm not sure)... Looks like this all happens in the background over several days. It'd be pretty cool if Panther got faster with extended use

quote:

---

Why is this necessary? So, it starts paging in and out in the middle of the night. It'll be done by morning, and the memory will be freed in due time. My concern is not so much about uptime () but about having to close stuff down and then starting it back up after login.

---

The problem with paging with this sort of routine is that you may page out all sorts of things that you actually want in memory in exchange for a bunch of files that you may or may not want in memory. Ideal might be to assign some sort of memory throttle for this routine e.g. Photoshop allows you to set a maximum % of available RAM to be used for the program. That allows you to prevent Photoshop from eating all your RAM.

If you could run a chron job background defragger and limit the process to 50% of available free RAM, that would prevent it from displacing stuff you are actually using.

Now, I'm not a programmer, but I would hope that OSX is getting smarter about what it pages out e.g. sets priorities for what code is less speed critical and pages that out. Has anyone seen any signs of this? My understanding is that OSX used to page out based on 'longest time since code has been used'. The problem with that is you may have some huge recent app or a memory leak that squeezes all sorts of speed critical code out to disk. Unless the file system prioritizes system files etc. over those three huge PDFs you recently opened but forgot to close, you can really slow down your performance.

These days, about the only reason I restart my box (aside from system updates) is because I've gone into swap. The computer simply never seems as fast once it's begun swapping. I'm convinced that's because the computer doesn't properly prioritize what to swap and ends up paging speed critical code back up.

-

Discord-

Thanks for the analysis. Sounds like the program may be doing exactly what I've been hoping for- intelligent defragging and strategic file placement. Yay!

quote:

---

Originally posted by iconmaster:
Doesn't the defragmentation happen on opening the file? As I understand it, saving wouldn't be necessary -- just a lot of opening and closing.

---

I'm not sure about that. I'm not sure how Apple implemented it.

I would think that defragging on open would defrag more files, however, it would allow for more possibility for errors because you're altering the file without (the user) making changes (when normally, a file's magnetic state wouldn't be altered until changes were made - by the user).

EDIT: I guess discord answered that question. Files are defragged on open, but only if they're small and really, really fragmented. So further optimization is possible.

quote:

---

Originally posted by discord:
The rest of the hotfile mechanism is interesting, and hasn't been discussed yet - it looks like over a period of time, OSX keeps track of which files have been use the most and moves them to the "hot band". This is already a bonus, since it'll defrag the files during the move, but I'm not sure what the "hot band" is (probably the fastest area on the physical hard disk, but I'm not sure)... Looks like this all happens in the background over several days. It'd be pretty cool if Panther got faster with extended use

---

I actually don't like this idea (except when VM is on an exclusive partition). I think that unused files should be moved onto the slowest part of the drive as opposed to moving often-used files to the fast part of the drive. Why? Because VM is used more than basically any file, and therefore performance of the system would be better if VM was always on the fastest part of the drive - and that can't happen if OS X is constantly piling files onto the fastest part of the drive.

Having said that, I have a separate partitions for VM, Scratch, OS+Apps, Files (non-AV), AV Files, and Installers + Updaters. So I would benefit greatly from the scheme described by discord. But my setup isn't typical. Then again, most users don't care.

Off Topic: I would just like to point out that new posters like discord are the reason that the Ach should not become subscriber only. discord, welcome to Ars Technica! (And welcome to all the other MacSlashers visiting!)

[This message was edited by TigerKR on October 30, 2003 at 02:41.]

quote:

---

Originally posted by discord:
It'd be pretty cool if Panther got faster with extended use

---

That's what I'm hoping too. (Rock on, Apple Computer.) Thanks for the great analysis, discord.

TigerKR: I think it's pretty clear from the comments:

         /*
         * On the first (non-busy) open of a fragmented
         * file attempt to de-frag it (if its less than 20MB).
         */

Edit: I see you were persuaded. But I'll leave my supporting evidence in 'cause it makes me look like I understand this stuff.

Are there any other hidden jewels in the Darwin 7 source code? I'm really wondering why Apple wouldn't document such a useful feature...