ABIS Infor - 2014-03
Hidden data on your file system?
Abstract
The story goes that one can hear hidden messages when playing backward some of the Led Zeppelin or The Beatles music. Likewise, rumours go that information can be hidden in your computer files. Or are these just urban legends?
Context
Each file system associates to a file -except for its content- also additional information, metadata, such as e.g. the date of last modification and the owner of the file. The file system HFS of the Apple MAC (and also its predecessor, MFS) even goes somewhat further. It provided since the mid 80s the possibility to attach arbitrary pieces of metadata (of arbitrary length) to a file. These metadata fragments are called resource forks or "alternate data streams", in contrast to the "ordinary" file data which is called the data fork. A typical application of a resource fork is the graphical icon (small bitmap) which should be displayed together with a file.
When the Windows NTFS file system was introduced (with Windows NT 3.1 in 1993), support for such "Alternate Data Streams" (ADS) was included, most likely to ease migrating from HFS to NTFS (and hence from Apple to Microsoft).
NTFS and ADS
The standard file system for MS-Windows, NTFS, allows attaching one or more data streams to a file. A first data stream contains the "Access Control List" (ACL, i.e., security information); all files (except for empty ones or shortcuts) of course also have a "content" data stream, or the "data fork". A shortcut additionally has a link data stream (containing the referring file name). But in addition NTFS allows to attach an arbitrary number of Alternate Data Streams (ADS) to a file, with arbitrary content.
To specify an ADS, its name must be attached, separated by a colon, to the file name. For example, to specify the ADS named "extra" for the file "C:\test.txt", the name "C:\test.txt:extra" has to be used. Plain and clear. Only... scarcely any Windows application supports this notation for the content of an ADS! Hence an ADS is completely invisible, both in Windows Explorer and with the DIR command. This makes the Alternate Data Streams de facto useless. Or not?
Use of ADS
Since Windows Vista, the new option "/R" to the DIR command-line instruction also shows all ADS in a directory. Specifically, the output of the instruction "DIR /R test.txt" could e.g. be as follows:
2014-02-26 17:48 600 test.txt
700 test.txt:extra:$DATA
1 File(s) 600 bytesClearly an extra line is shown (as compared to "DIR" without "/R"), but the summary still states "1 File, 600 bytes" (i.e., the size of the "ordinary" data stream); just as what Windows Explorer shows, by the way. The extra 700 bytes are thus effectively hidden! (Observe the additional ":$DATA" which is always shown by DIR/R but which is not relevant.)
These 700 bytes certainly take up disk space, but if we cannot display them, or create, or modify, or remove, they are of course useless. Well, there is only one way to do this on a "standard" Windows system, i.e., one without additional software. It requires using the command line, just like for DIR. So we first need a short intermezzo on this non-graphical interface of Windows, also called the "DOS box".
cmd.exe and redirecting data streams
When starting up the Windows application cmd.exe, a black window opens, showing a prompt of the form C:\xxx> (where xxx will be replaced by the name of the active directory). Type an instruction after this prompt (e.g. "DIR /R") followed by the ENTER key, and the result of the command will appear in the same black window.
Now type exactly the same command, but before you press the ENTER key add the following text (without the quotes): " > test.txt" . The result of the command no longer appears on the screen, but instead is "redirected" to the file test.txt, which will be either created or overwritten as a consequence of your last action. There are also some instruction of cmd.exe which would rather like to read instead of writing to the screen. A good example of this is the instruction "more": to inspect the content of a file (on the screen), "more" needs a data stream, or otherwise said the content of a file. To display the content of the file "test.txt", one may use the instruction "more < test.txt". So the "<" notation will redirect the input stream towards an instruction from a file, while ">" redirects the output stream to a (new) file. Instructions which read as well as write, may of course use both ">" and "<".
It turns out that only within cmd.exe, and only at the right hand side of either "<" or ">", the ":" notation is understood. (A few other commands understand the ADS notation, like e.g. Notepad, but only when started up from within cmd.exe)
ADS manipulations, a summary
1. How do I create a new ADS named extra for the file test.txt?
more < somefile.txt > test.txt:extra
This copies the content of somefile.txt to the ADS named extra of test.txt (and thereby replaces its old content). Note that the file test.txt need not exist for this instruction to succeed; in that case, it seems like an empty file test.txt has been created (file size 0 bytes), which strictly speaking isn't true: there is no data fork, only an ADS!
2. How do I display the content of the ADS named extra of the file test.txt?
more < test.txt:extra
3. How do I delete the ADS named extra of test.txt?
Most easily, just "empty" its content. For this purpose, start by creating an empty file, e.g. by issuing the instruction
"copy NUL: empty.txt", followed by
more < empty.txt > test.txt:extra
Or even better: first rename test.txt to test2.txt with "rename test.txt test2.txt". This "new" file test2.txt is not really new: it is the old file of which just the "name" metadata has been changed. It still has an ADS, now called "test2.txt:extra". Next copy the content of test2.txt to test.txt using
"copy test2.txt test.txt". The copy instruction does not support ADS, hence no test.txt:extra got created. Only the "main" data stream has been copied. Finally remove test2.txt: "del test2.txt".
It should be mentioned that also directories (like e.g. C:\Windows) can have one or more ADS. Such an ADS will be very difficult to remove, though, if the directory is a system directory ...
Conclusion
The so-called "Alternate Data Streams" (ADS) of NTFS are a well saved Windows secret, which allows you to store information on your hard disk without anyone knowing about their existence (nor content). On the other hand, none of the "ordinary" Windows applications supports these file parasites, so be very careful when e.g. copying, dragging/dropping, zipping, compressing, converting, ... a file to which a valuable ADS is attached!
References: