Why do USB drives need to be "ejected"

[Disch]

In another thread I am complaining about an experience I was having with a USB drive.

Basically the drive was not properly "ejected" and as a result there's potential file corruption due to incomplete writes.

Of course the proper way to remove a USB drive is to "eject" it through the software before physically disconnecting it from the computer.


Does anybody else find this whole concept archaic and unnecessary?

Is it really such a difficult hardware problem to flush writes to the drive right away?

It seems silly that it's still required. Especially considering how fast external drives have gotten.

[peppers]

I am more concerned why e-SATA dose not have this feature, I am nervous every time I unplug my 2 tb external, running a file recovery on it would be a pain in the ass.

I agree though, I don't know why Windows has to do a bunch of unnecessary useless bullshit wrights to external devices, if it was not for those its as simple as not unplugging it wile your transferring shit.

[Dwedit]

Sysinternals Sync lets you commit writes.

[Disch]

I agree though, I don't know why Windows has to do a bunch of unnecessary useless bullshit wrights to external devices, if it was not for those its as simple as not unplugging it wile your transferring shit.

It is just Windows that has the problem?

AFAIK, the problem exists on Linux as well (and possibly Mac?)

[peppers]

Are you sure this is an issue in Linux?
its true Linux gives you the option of un-mounting any partition but I think it pretty much leaves drives alone unless you are at the very least have a directory open. I remember reading something about it de-fragmenting its drives on the fly and I don't know exactly how that works but I would assume its done during wrights to the hard drive you make.

[blargg]

How does the average user know that no program is currently accessing the USB device when he goes to remove it? The filesystem could be currently writing to the file structures. Even if it's not, having possible removal at any time could upset programs which have open files. This same issue of course occurred with floppy drives, which is one reason Macs had no hardware eject button.

[BMF54123]

Sysinternals Sync lets you commit writes.

I have a batch file sitting in my Quick Launch tray that runs Sync on every removable drive in the system when I click it. I did this primarily to work around a stupid design flaw in my internal x-in-1 card reader: attempting to eject any of the drives actually "ejects" the reader itself, rendering it useless until I reboot (since it's plugged into the motherboard's USB header, I can't just yank it and plug it back in). But now, when I'm done with a card, I can click the batch file, wait for the command window to disappear, and safely remove the card without fear of data loss.

I think what infuriates me even more than the ejection process itself is when Windows unhelpfully tells you "The device [name] cannot be stopped right now. Try stopping it again later." This is just about the laziest don't-give-a-sh*t error message I have ever seen Windows spew out, and it makes me angry every time I see it. So I'm supposed to just sit on my thumbs for some arbitrary length of time and pray to God Windows eventually unhooks its barbs from my files? What really gets me about this is, more often than not, the error pops up because some poorly-coded program (or even Windows itself) left a rogue file handle open, leading the OS to believe the drive is still in use. I've actually had to run Process Explorer and manually hunt down zombie handles on more than one occasion because they persisted across reboots! How the hell does that work?

arrgh rant fume gripe moan

[blargg]

On my OS X 10.3.9 machine, I sometimes drag the camera's USB drive icon to the trash, wait for its icon to disappear, wait a few seconds more, then unplug the camera, and the damn OS still complains to me that the device has been unexpectedly disconnected. Whatever, I say.

[tepples]

I think what infuriates me even more than the ejection process itself is when Windows unhelpfully tells you "The device [name] cannot be stopped right now. Try stopping it again later."

What this error message means is that some program has something open on this drive. This could be an open file (fopen()), an open directory (findfirst() or opendir()), as you pointed out, or even just the current working directory (chdir()). If I've used an app for a while, one of the directory handles used by the Open or Save As dialog box might be pointed at a removable drive. Closing and reopening the app will free up any objects that the app has open on that drive.

I've seldom had a drive fail to stop after closing all applications. And it has never happened after logging out all users and logging back in (which restarts Windows Explorer).

What really gets me about this is, more often than not, the error pops up because some poorly-coded program (or even Windows itself) left a rogue file handle open

After the user has chosen a file to open, what is a program supposed to do with directory handles owned by the Open and Save As dialog boxes? I'd guess the operating system would have to provide a way for an application to mark a handle as eligible for surprise closure by a user.

leading the OS to believe the drive is still in use. I've actually had to run Process Explorer and manually hunt down zombie handles on more than one occasion because they persisted across reboots! How the hell does that work?

The file handle doesn't persist across reboots, to my knowledge. Otherwise, something would screw up if you shut down, yank hardware, and start the machine. What might be happening is that something listed in ...\Windows\CurrentVersion\Run or in Control Panel > Administrative Tools > Services is opening the file before you can get to the taskbar to stop the drive.

[After correctly unmounting a camera's storage on Mac OS X,] the damn OS still complains to me that the device has been unexpectedly disconnected.

That might happen with a composite device. A camera might include, for example, an internal storage (which you unmounted with the trash can) and a live video input device (for use as a webcam). You might have to close iMovie or whatever to free the video input handle.

[koitsu]

In another thread I am complaining about an experience I was having with a USB drive. Basically the drive was not properly "ejected" and as a result there's potential file corruption due to incomplete writes. Of course the proper way to remove a USB drive is to "eject" it through the software before physically disconnecting it from the computer.

Does anybody else find this whole concept archaic and unnecessary? Is it really such a difficult hardware problem to flush writes to the drive right away? It seems silly that it's still required. Especially considering how fast external drives have gotten.

1) The process is unnecessary assuming drive write caching is turned off (more on that in a moment), and you don't have any applications (hint hint: anti-virus or Windows indexing services) which have files open on the device prior to removal. If any files are open, or write caching is enabled, then you will see said error.

2) Flushing writes to the disk every time can be done indirectly by disabling write caching. On 2K/XP, this is the default. So if you're seeing this message on 2K/XP, then you have a process of some kind that has an open file on the USB device. You can use SysInternals software to track down what that is, but it'll be a battle.

3) The speed of the device (disk, flash, etc.) has absolutely nothing to do with the problem.

If you figure out a way, via software, to pre-emptively "guess" when a user is about to physically yank a device off a bus, let me know. :-)

[koitsu]

I am more concerned why e-SATA dose not have this feature, I am nervous every time I unplug my 2 tb external, running a file recovery on it would be a pain in the ass.

Let me ask you something. What's more important to you: a major transfer speed increase (due to write caching being enabled), or the ability to yank the device off the bus at any time? If the latter, disable write caching on the device.

Otherwise, see my post to Disch, re: let me know if you have a way via software to read the minds of users.

I agree though, I don't know why Windows has to do a bunch of unnecessary useless bullshit wrights to external devices, if it was not for those its as simple as not unplugging it wile your transferring shit.

This isn't a Windows problem. Every operating system has the exact same issue, and every operating system has solved it in the exact same two ways that Windows has (disable write caching on the device, or require the user to use an application/applet that flushes pending writes to the device prior to device unmount (not the same thing as physical removal)).

You can blame Windows for things like the Indexing Service, and maybe even anti-virus software or background services that do something with underlying files or file handles on a device. But the problem in question is really not "the fault of Windows".

[Disch]

Are you sure this is an issue in Linux?

Yes, I'm pretty sure. At least it was with my sister's drive.

I copied files to it, then unplugged the drive (a good few seconds after the copy was complete just to be sure), but the files were corrupt when I got to my sister's place.

3) The speed of the device (disk, flash, etc.) has absolutely nothing to do with the problem.

I figured the problem might be related to write being "complete" when they're not really complete. Disk writes aren't instantaneous, and when you save a file or something, it doesn't necessarily save directly to the disk if the disk is busy, but might instead be put in a cache and queued to be saved when the drive has time.

If drives are faster, this is less of a problem, as the queues will be emptied faster, and writes commited sooner.

If you figure out a way, via software, to pre-emptively "guess" when a user is about to physically yank a device off a bus, let me know. :-)

You shouln't need to know when the user is going to remove the device, is my point.

If you're reading something from the drive and it disappears... oh well. Give an error to the user that your read failed. It's not the end of the world, and wouldn't cause any problems on the drive.

If you're writing something to the drive and it disappears, that's a big problem -- but programs typically make it clear that they're writing to the drive while they're writing. Like when you copy files to the drive, a progress box pops up. Or if you save something to it from another program the program typically stalls until the save is complete (with an optional progress bar). Hopefully the user will have enough sense not to remove the drive during such a time.

The only way it'd be a problem is if the program is doing constant little mini-writes. Like if a program has a config file or something open that they need to constantly read/write to. But putting such a config file on a flash drive is either bad design of the program, or a very poor decision by the user.

[UncleSporky]

leading the OS to believe the drive is still in use. I've actually had to run Process Explorer and manually hunt down zombie handles on more than one occasion because they persisted across reboots! How the hell does that work?

The file handle doesn't persist across reboots, to my knowledge. Otherwise, something would screw up if you shut down, yank hardware, and start the machine. What might be happening is that something listed in ...\Windows\CurrentVersion\Run or in Control Panel > Administrative Tools > Services is opening the file before you can get to the taskbar to stop the drive.

Is this at all related to the sort of problems I rarely get where I try to delete a folder but I can't because it's "in use," and it always persists across reboots, and I have to use Unlocker to release the OS's grip on it?

It seems to me that a file on the flash drive could also get stuck the same way, right? After a reboot, Windows checks its list of locked files and goes "whoa, nope, there's a file I'm still using on there."

[tepples]

If you're reading something from the drive and it disappears... oh well. Give an error to the user that your read failed.

A lot of applications on which PC users depend are poorly written and would crash or worse when trying to read from a disappeared file. This has to be fixed at the application level, but much of the time, either an app is no longer maintained or its maintainer would just discard defect reports related to disappearing media. When apps crash, enough users have a tendency to blame it not on the app but on the operating system, which is one reason why operating systems chastise the user for surprise-removing a device containing a mounted file system: to remind the user that application misbehaviors are not always the operating system's fault.

The only way it'd be a problem is if the program is doing constant little mini-writes.

And in a lot of cases, this is the case. For example, Firefox on my Windows machine appears to compact its history search index (places.sqlite) in a thread in the background. I know it's happening because I can hear my hard drive seeking, but Firefox doesn't put an icon in the browser window to show that this is happening. Imagine an e-mail client that spawns a thread to check for new messages every five minutes. Unless it has a countdown timer, there's no way to prevent the user from yanking a drive half a second after a download of new messages has started.

But putting such a config file on a flash drive is either bad design of the program, or a very poor decision by the user.

Some apps have no persistent storage other than flash. This includes games on handhelds and game consoles from 1995 or later: "Now saving. Do not touch the memory card or turn the power off." It also includes PC apps capable of being installed on removable media.

Is this at all related to the sort of problems I rarely get where I try to delete a folder but I can't because it's "in use," and it always persists across reboots, and I have to use Unlocker to release the OS's grip on it?

I think that's file system corruption. As I understand it, a quick chkdsk should clear this up: it warns the user, force-unmounts the drive, and turns off all files' "in use" bits. I understand that newer applications are capable of determining what other process has locked a file, which means it must be a new API introduced in a new Windows version.

[Disch]

And in a lot of cases, this is the case. For example, Firefox on my Windows machine appears to compact its history search index (places.sqlite) in a thread in the background.

Yes... but on your removable drive? Not likely unless you installed Firefox on your removable drive, which would probably be a foolhardy move.

And if you did, you would be expected to close Firefox before removing the drive (see next point)

Some apps have no persistent storage other than flash. This includes games on handhelds and game consoles from 1995 or later: "Now saving. Do not touch the memory card or turn the power off." It also includes PC apps capable of being installed on removable media.

Right. But the user could be expected to close any program that's running off of the drive before they remove the drive.

Let's assume the user has common sense here. If the user lacks common sense there's obviously nothing that can be done. But for those of us that do have common sense, for us to still have this limitation is a big inconvenience.

1) I'm not running any executables off the drive
2) I'm not saving any files to the drive
3) I'm not copying any files to the drive
4) I'm not doing any other kind of task that obviously is writing (like defragging/formatting)

If the above are true, I should be able to yank the drive at any time.