Misplaced Pages

#29970

72-501: Gzip is a file format and a software application used for file compression and decompression . The program was created by Jean-loup Gailly and Mark Adler as a free software replacement for the compress program used in early Unix systems, and intended for use by GNU (from which the "g" of gzip is derived). Version 0.1 was first publicly released on 31 October 1992, and version 1.0 followed in February 1993. The decompression of

144-434: A byte frequency distribution to build the representative models for file type and use any statistical and data mining techniques to identify file types. There are several types of ways to structure data in a file. The most usual ones are described below. Earlier file formats used raw data formats that consisted of directly dumping the memory images of one or more structures into the file. This has several drawbacks. Unless

216-408: A company logo may be needed both in .eps format (for publishing) and .png format (for web sites). With the extensions visible, these would appear as the unique filenames: " CompanyLogo.eps " and " CompanyLogo.png ". On the other hand, hiding the extensions would make both appear as " CompanyLogo ", which can lead to confusion. Hiding extensions can also pose a security risk. For example,

288-520: A few bytes long. The metadata contained in a file header are usually stored at the start of the file, but might be present in other areas too, often including the end, depending on the file format or the type of data contained. Character-based (text) files usually have character-based headers, whereas binary formats usually have binary headers, although this is not a rule. Text-based file headers usually take up more space, but being human-readable, they can easily be examined by using simple software such as

360-448: A file based on the end of its name, more specifically the letters following the final period. This portion of the filename is known as the filename extension . For example, HTML documents are identified by names that end with .html (or .htm ), and GIF images by .gif . In the original FAT file system , file names were limited to an eight-character identifier and a three-character extension, known as an 8.3 filename . There are

432-401: A file format is to use information regarding the format stored inside the file itself, either information meant for this purpose or binary strings that happen to always be in specific locations in files of some formats. Since the easiest place to locate them is at the beginning, such area is usually called a file header when it is greater than a few bytes , or a magic number if it is just

504-416: A file unusable (or "lose" it) by renaming it incorrectly. This led most versions of Windows and Mac OS to hide the extension when listing files. This prevents the user from accidentally changing the file type, and allows expert users to turn this feature off and display the extensions. Hiding the extension, however, can create the appearance of two or more identical filenames in the same folder. For example,

576-401: A formal specification document, letting precedent set by other already existing programs that use the format define the format via how these existing programs use it. If the developer of a format does not publish free specifications, another developer looking to utilize that kind of file must either reverse engineer the file to find out how to read it or acquire the specification document from

648-402: A gzip replacement. It produces considerably smaller files (especially for source code and other structured text), but at the cost of memory and processing time (up to a factor of 4). AdvanceCOMP, Zopfli , libdeflate and 7-Zip can produce gzip-compatible files, using an internal DEFLATE implementation with better compression ratios than gzip itself—at the cost of more processor time compared to

720-399: A hierarchical structure, known as a conformance hierarchy. Thus, public.png conforms to a supertype of public.image , which itself conforms to a supertype of public.data . A UTI can exist in multiple hierarchies, which provides great flexibility. In addition to file formats, UTIs can also be used for other entities which can exist in macOS, including: In IBM OS/VS through z/OS ,

792-442: A limited number of three-letter extensions, which can cause a given extension to be used by more than one program. Many formats still use three-character extensions even though modern operating systems and application programs no longer have this limitation. Since there is no standard list of extensions, more than one format can use the same extension, which can confuse both the operating system and users. One artifact of this approach

SECTION 10

#1732855695030

864-414: A malicious user could create an executable program with an innocent name such as " Holiday photo.jpg.exe ". The " .exe " would be hidden and an unsuspecting user would see " Holiday photo.jpg ", which would appear to be a JPEG image, usually unable to harm the machine. However, the operating system would still see the " .exe " extension and run the program, which would then be able to cause harm to

936-408: A particular file's format, with each approach having its own advantages and disadvantages. Most modern operating systems and individual applications need to use all of the following approaches to read "foreign" file formats, if not work with them completely. One popular method used by many operating systems, including Windows , macOS , CP/M , DOS , VMS , and VM/CMS , is to determine the format of

1008-585: A single tar archive (also called tarball ), and then compressing that archive with gzip. The final compressed file usually has the extension .tar.gz or .tgz . gzip is not to be confused with the ZIP archive format, which also uses DEFLATE . The ZIP format can hold collections of files without an external archiver, but is less compact than compressed tarballs holding the same data, because it compresses files individually and cannot take advantage of redundancy between files ( solid compression ). The gzip file format

1080-441: A sorted index). Also, data must be read from the file itself, increasing latency as opposed to metadata stored in the directory. Where file types do not lend themselves to recognition in this way, the system must fall back to metadata. It is, however, the best way for a program to check if the file it has been told to process is of the correct format: while the file's name or metadata may be altered independently of its content, failing

1152-431: A specialized editor or IDE . However, this feature was often the source of user confusion, as which program would launch when the files were double-clicked was often unpredictable. RISC OS uses a similar system, consisting of a 12-bit number which can be looked up in a table of descriptions—e.g. the hexadecimal number FF5 is "aliased" to PoScript , representing a PostScript file. A Uniform Type Identifier (UTI)

1224-485: A text editor or a hexadecimal editor. As well as identifying the file format, file headers may contain metadata about the file and its contents. For example, most image files store information about image format, size, resolution and color space , and optionally authoring information such as who made the image, when and where it was made, what camera model and photographic settings were used ( Exif ), and so on. Such metadata may be used by software reading or interpreting

1296-495: A value in a company/standards organization database), and the 2 following digits categorize the type of file in hexadecimal . The final part is composed of the usual filename extension of the file or the international standard number of the file, padded left with zeros. For example, the PNG file specification has the FFID of 000000001-31-0015948 where 31 indicates an image file, 0015948

1368-630: A way of identifying what type of file was attached to an e-mail , independent of the source and target operating systems. MIME types identify files on BeOS , AmigaOS 4.0 and MorphOS , as well as store unique application signatures for application launching. In AmigaOS and MorphOS, the Mime type system works in parallel with Amiga specific Datatype system. There are problems with the MIME types though; several organizations and people have created their own MIME types without registering them properly with IANA, which makes

1440-428: A well-designed magic number test is a pretty sure sign that the file is either corrupt or of the wrong type. On the other hand, a valid magic number does not guarantee that the file is not corrupt or is of a correct type. So-called shebang lines in script files are a special case of magic numbers. Here, the magic number is human-readable text that identifies a specific command interpreter and options to be passed to

1512-610: Is Zopfli . It achieves gzip-compatible compression using more exhaustive algorithms, at the expense of compression time required. It does not affect decompression time. pigz , written by Mark Adler, is compatible with gzip and speeds up compression by using all available CPU cores and threads. Data in blocks prior to the first damaged part of the archive is usually fully readable. Data from blocks not demolished by damage that are located afterward may be recoverable through difficult workarounds. The tar utility included in most Linux distributions can extract .tar.gz files by passing

SECTION 20

#1732855695030

1584-617: Is a software library used for data compression as well as a data format. zlib was written by Jean-loup Gailly and Mark Adler and is an abstraction of the DEFLATE compression algorithm used in their gzip file compression program. zlib is also a crucial component of many software platforms, including Linux , macOS , and iOS . It has also been used in gaming consoles such as the PlayStation 4 , PlayStation 3 , Wii U , Wii , Xbox One and Xbox 360 . The first public version of Zlib, 0.9,

1656-528: Is a method used in macOS for uniquely identifying "typed" classes of entities, such as file formats. It was developed by Apple as a replacement for OSType (type & creator codes). The UTI is a Core Foundation string , which uses a reverse-DNS string. Some common and standard types use a domain called public (e.g. public.png for a Portable Network Graphics image), while other domains can be used for third-party types (e.g. com.adobe.pdf for Portable Document Format ). UTIs can be defined within

1728-407: Is a risk that the file format can be misinterpreted. It may even have been badly written at the source. This can result in corrupt metadata which, in extremely bad cases, might even render the file unreadable. A more complex example of file headers are those used for wrapper (or container) file formats. One way to incorporate file type metadata, often associated with Unix and its derivatives,

1800-459: Is advisable to write regular full-flush points. Today, zlib is something of a de facto standard , to the point that zlib and DEFLATE are often used interchangeably in standards documents, with thousands of applications relying on it for compression, either directly or indirectly. These include: zlib is also used in many embedded devices, such as the Apple iPhone and Sony PlayStation 3 , because

1872-467: Is also not to be confused with that of the compress utility, based on LZW, with extension .Z ; however, the gunzip utility is able to decompress .Z files. Various implementations of the program have been written. The most commonly known is the GNU Project's implementation using Lempel-Ziv coding (LZ77). OpenBSD 's version of gzip is actually the compress program, to which support for the gzip format

1944-446: Is an extensible scheme of persistent, unique, and unambiguous identifiers for file formats, which has been developed by The National Archives of the UK as part of its PRONOM technical registry service. PUIDs can be expressed as Uniform Resource Identifiers using the info:pronom/ namespace. Although not yet widely used outside of the UK government and some digital preservation programs,

2016-424: Is small, and/or that chunks do not contain other chunks; many formats do not impose those requirements. The information that identifies a particular "chunk" may be called many different things, often terms including "field name", "identifier", "label", or "tag". The identifiers are often human-readable, and classify parts of the data: for example, as a "surname", "address", "rectangle", "font name", etc. These are not

2088-404: Is smaller than the gzip wrapper (RFC 1952), as the latter stores a file name and other file system information. As of September 2018 , zlib only supports one algorithm, called DEFLATE , which uses a combination of a variation of LZ77 (Lempel–Ziv 1977) and Huffman coding . This algorithm provides good compression on a wide variety of data with minimal use of system resources. This is also

2160-427: Is that the system can easily be tricked into treating a file as a different format simply by renaming it — an HTML file can, for instance, be easily treated as plain text by renaming it from filename.html to filename.txt . Although this strategy was useful to expert users who could easily understand and manipulate this information, it was often confusing to less technical users, who could accidentally make

2232-637: Is the standard number and 000000001 indicates the International Organization for Standardization (ISO). Another less popular way to identify the file format is to examine the file contents for distinguishable patterns among file types. The contents of a file are a sequence of bytes and a byte has 256 unique permutations (0–255). Thus, counting the occurrence of byte patterns that is often referred to as byte frequency distribution gives distinguishable patterns to identify file types. There are many content-based file type identification schemes that use

gzip - Misplaced Pages Continue

2304-475: Is to store a "magic number" inside the file itself. Originally, this term was used for a specific set of 2-byte identifiers at the beginnings of files, but since any binary sequence can be regarded as a number, any feature of a file format which uniquely distinguishes it can be used for identification. GIF images, for instance, always begin with the ASCII representation of either GIF87a or GIF89a , depending upon

2376-439: Is using the library to always compress specific types of data, then using a specific strategy may improve compression and performance. For example, if the data contain long lengths of repeated bytes, the run-length encoding (RLE) strategy may give good results at a higher speed. For general data, the default strategy is preferred. Errors in compressed data may be detected and skipped. Further, if "full-flush" points are written to

2448-521: The z option, e.g., tar -zxf file.tar.gz , where -z instructs decompression, -x means extraction, and -f specifies the name of the compressed archive file to extract from. Optionally, -v ( verbose ) lists files as they are being extracted. zlib is an abstraction of the DEFLATE algorithm in library form which includes support both for the gzip file format and a lightweight data stream format in its API. The zlib stream format, DEFLATE, and

2520-629: The GIF file format required the use of a patented algorithm, and though the patent owner did not initially enforce their patent, they later began collecting royalty fees . This has resulted in a significant decrease in the use of GIFs, and is partly responsible for the development of the alternative PNG format. However, the GIF patent expired in the US in mid-2003, and worldwide in mid-2004. Different operating systems have traditionally taken different approaches to determining

2592-455: The Ogg format can act as a container for different types of multimedia including any combination of audio and video , with or without text (such as subtitles ), and metadata . A text file can contain any stream of characters, including possible control characters , and is encoded in one of various character encoding schemes . Some file formats, such as HTML , scalable vector graphics , and

2664-408: The gzip format can be implemented as a streaming algorithm , an important feature for Web protocols , data interchange and ETL (in standard pipes ) applications. gzip is based on the DEFLATE algorithm, which is a combination of LZ77 and Huffman coding . DEFLATE was intended as a replacement for LZW and other patent -encumbered data compression algorithms which, at the time, limited

2736-468: The source code of computer software are text files with defined syntaxes that allow them to be used for specific purposes. File formats often have a published specification describing the encoding method and enabling testing of program intended functionality. Not all formats have freely available specification documents, partly because some developers view their specification documents as trade secrets , and partly because other developers never author

2808-467: The ASCII representation formed a sequence of meaningful characters, such as an abbreviation of the application's name or the developer's initials. For instance a HyperCard "stack" file has a creator of WILD (from Hypercard's previous name, "WildCard") and a type of STAK . The BBEdit text editor has a creator code of R*ch referring to its original programmer, Rich Siegel . The type code specifies

2880-509: The PUID scheme does provide greater granularity than most alternative schemes. MIME types are widely used in many Internet -related applications, and increasingly elsewhere, although their usage for on-disc type information is rare. These consist of a standardised system of identifiers (managed by IANA ) consisting of a type and a sub-type , separated by a slash —for instance, text/html or image/gif . These were originally intended as

2952-715: The VSAM catalog (prior to ICF catalogs ) and the VSAM Volume Record in the VSAM Volume Data Set (VVDS) (with ICF catalogs) identifies the type of VSAM dataset. In IBM OS/360 through z/OS , a format 1 or 7 Data Set Control Block (DSCB) in the Volume Table of Contents (VTOC) identifies the Dataset Organization ( DSORG ) of the dataset described by it. The HPFS , FAT12, and FAT16 (but not FAT32) filesystems allow

gzip - Misplaced Pages Continue

3024-514: The algorithm used in the Zip archive format . The header makes allowance for other algorithms, but none are currently implemented. zlib provides facilities for control of processor and memory use. A compression level value may be supplied that trades speed for compression. There are also facilities for conserving memory, useful in restricted memory environments, such as some embedded systems. The compression can be optimized for specific types of data. If one

3096-406: The amount (ratio) of compression. There is no limit to the length of data that can be compressed or decompressed. Repeated calls to the library allow an unlimited number of blocks of data to be handled. Some ancillary code (counters) may suffer from overflow for long data streams, but this does not affect the actual compression or decompression. When compressing a long (or infinite) data stream, it

3168-442: The appropriate icons, but these will be located in different places on the storage medium thus taking longer to access. A folder containing many files with complex metadata such as thumbnail information may require considerable time before it can be displayed. If a header is binary hard-coded such that the header itself needs complex interpretation in order to be recognized, especially for metadata content protection's sake, there

3240-450: The command interpreter. Another operating system using magic numbers is AmigaOS , where magic numbers were called "Magic Cookies" and were adopted as a standard system to recognize executables in Hunk executable file format and also to let single programs, tools and utilities deal automatically with their saved data files, or any other kind of file types when saving and loading data. This system

3312-403: The compressed stream, then corrupt data can be skipped, and the decompression will resynchronize at the next flush point—although no error recovery of the corrupt data is provided. Full-flush points are useful for large data streams on unreliable channels, where some data loss is unimportant, such as in some multimedia applications. However, creating many flush points can affect the speed as well as

3384-431: The computer. The same is true with files with only one extension: as it is not shown to the user, no information about the file can be deduced without explicitly investigating the file. To further trick users, it is possible to store an icon inside the program, in which case some operating systems' icon assignment for the executable file ( .exe ) would be overridden with an icon commonly used to represent JPEG images, making

3456-419: The data must be entirely parsed by applications. On Unix and Unix-like systems, the ext2 , ext3 , ext4 , ReiserFS version 3, XFS , JFS , FFS , and HFS+ filesystems allow the storage of extended attributes with files. These include an arbitrary list of "name=value" strings, where the names are unique and a value can be accessed through its related name. The PRONOM Persistent Unique Identifier (PUID)

3528-430: The destination, the single file received has to be unzipped by a compatible utility to be useful. The problems of handling metadata are solved this way using zip files or archive files. The Mac OS ' Hierarchical File System stores codes for creator and type as part of the directory entry for each file. These codes are referred to as OSTypes. These codes could be any 4-byte sequence but were often selected so that

3600-435: The file during the loading process and afterwards. File headers may be used by an operating system to quickly gather information about a file without loading it all into memory, but doing so uses more of a computer's resources than reading directly from the directory information. For instance, when a graphic file manager has to display the contents of a folder, it must read the headers of many files before it can display

3672-448: The file format's definition. Throughout the 1970s, many programs used formats of this general kind. For example, word-processors such as troff , Script , and Scribe , and database export files such as CSV . Electronic Arts and Commodore - Amiga also used this type of file format in 1985, with their IFF (Interchange File Format) file format. A container is sometimes called a "chunk" , although "chunk" may also imply that each piece

SECTION 50

#1732855695030

3744-809: The file, each of which is a string, such as "Plain Text" or "HTML document". Thus a file may have several types. The NTFS filesystem also allows storage of OS/2 extended attributes, as one of the file forks , but this feature is merely present to support the OS/2 subsystem (not present in XP), so the Win32 subsystem treats this information as an opaque block of data and does not use it. Instead, it relies on other file forks to store meta-information in Win32-specific formats. OS/2 extended attributes can still be read and written by Win32 programs, but

3816-543: The form of headers and trailers. Still, the gzip format is sometimes recommended over zlib because Internet Explorer does not implement the standard correctly and cannot handle the zlib format as specified in RFC 1950. zlib DEFLATE is used internally by the Portable Network Graphics (PNG) format. Since the late 1990s, bzip2 , a file compression utility based on a block-sorting algorithm, has gained some popularity as

3888-471: The format of the file, while the creator code specifies the default program to open it with when double-clicked by the user. For example, the user could have several text files all with the type code of TEXT , but each open in a different program, due to having differing creator codes. This feature was intended so that, for example, human-readable plain-text files could be opened in a general-purpose text editor, while programming or HTML code files would open in

3960-463: The format will be identified correctly, and can often determine more precise information about the file. Since reasonably reliable "magic number" tests can be fairly complex, and each file must effectively be tested against every possibility in the magic database, this approach is relatively inefficient, especially for displaying large lists of files (in contrast, file name and metadata-based methods need to check only one piece of data, and match it against

4032-586: The format's developers for a fee and by signing a non-disclosure agreement . The latter approach is possible only when a formal specification document exists. Both strategies require significant time, money, or both; therefore, file formats with publicly available specifications tend to be supported by more programs. Patent law, rather than copyright , is more often used to protect a file format. Although patents for file formats are not directly permitted under US law, some formats encode data using patented algorithms . For example, prior to 2004, using compression with

4104-564: The gzip file format were standardized respectively as RFC 1950, RFC 1951, and RFC 1952. The gzip format is used in HTTP compression , a technique used to speed up the sending of HTML and other content on the World Wide Web . It is one of the three standard formats for HTTP compression as specified in RFC 2616. This RFC also specifies a zlib format (called "DEFLATE"), which is equal to the gzip format except that gzip adds eleven bytes of overhead in

4176-515: The main data and the name, but is also less portable than either filename extensions or "magic numbers", since the format has to be converted from filesystem to filesystem. While this is also true to an extent with filename extensions— for instance, for compatibility with MS-DOS 's three character limit— most forms of storage have a roughly equivalent definition of a file's data and name, but may have varying or no representation of further metadata. Note that zip files or archive files solve

4248-430: The memory images also have reserved spaces for future extensions, extending and improving this type of structured file is very difficult. It also creates files that might be specific to one platform or programming language (for example a structure containing a Pascal string is not recognized as such in C ). On the other hand, developing tools for reading and writing these types of files is very simple. The limitations of

4320-416: The problem of handling metadata. A utility program collects multiple files together along with metadata about each file and the folders/directories they came from all within one new file (e.g. a zip file with extension .zip ). The new file is also compressed and possibly encrypted, but now is transmissible as a single file across operating systems by FTP transmissions or sent by email as an attachment. At

4392-447: The program look like an image. Extensions can also be spoofed: some Microsoft Word macro viruses create a Word file in template format and save it with a .doc extension. Since Word generally ignores extensions and looks at the format of the file, these would open as templates, execute, and spread the virus. This represents a practical problem for Windows systems where extension-hiding is turned on by default. A second way to identify

SECTION 60

#1732855695030

4464-747: The reference implementation. Research published in 2023 showed that simple lossless compression techniques such as gzip could be combined with a k-nearest-neighbor classifier to create an attractive alternative to deep neural networks for text classification in natural language processing . This approach has been shown to equal and in some cases outperform conventional approaches such as BERT due to low resource requirements, e.g. no requirement for GPU hardware. File format Some file formats are designed for very particular types of data: PNG files, for example, store bitmapped images using lossless data compression . Other file formats, however, are designed for storage of several different types of data:

4536-815: The same thing as identifiers in the sense of a database key or serial number (although an identifier may well identify its associated data as such a key). With this type of file structure, tools that do not know certain chunk identifiers simply skip those that they do not understand. Depending on the actual meaning of the skipped data, this may or may not be useful ( CSS explicitly defines such behavior). This concept has been used again and again by RIFF (Microsoft-IBM equivalent of IFF), PNG, JPEG storage, DER ( Distinguished Encoding Rules ) encoded streams and files (which were originally described in CCITT X.409:1984 and therefore predate IFF), and Structured Data Exchange Format (SDXF) . Indeed, any data format must somehow identify

4608-609: The significance of its component parts, and embedded boundary-markers are an obvious way to do so: This is another extensible format, that closely resembles a file system ( OLE Documents are actual filesystems), where the file is composed of 'directory entries' that contain the location of the data within the file itself as well as its signatures (and in certain cases its type). Good examples of these types of file structures are disk images , executables , OLE documents TIFF , libraries . Zlib zlib ( / ˈ z iː l ɪ b / or " zeta -lib", / ˈ z iː t ə ˌ l ɪ b / )

4680-583: The standard to which they adhere. Many file types, especially plain-text files, are harder to spot by this method. HTML files, for example, might begin with the string <html> (which is not case sensitive), or an appropriate document type definition that starts with <!DOCTYPE html , or, for XHTML , the XML identifier, which begins with <?xml . The files can also begin with HTML comments, random text, or several empty lines, but still be usable HTML. The magic number approach offers better guarantees that

4752-438: The storage of "extended attributes" with files. These comprise an arbitrary set of triplets with a name, a coded type for the value, and a value, where the names are unique and values can be up to 64 KB long. There are standardized meanings for certain types and names (under OS/2 ). One such is that the ".TYPE" extended attribute is used to determine the file type. Its value comprises a list of one or more file types associated with

4824-414: The unstructured formats led to the development of other types of file formats that could be easily extended and be backward compatible at the same time. In this kind of file structure, each piece of data is embedded in a container that somehow identifies the data. The container's scope can be identified by start- and end-markers of some kind, by an explicit length field somewhere, or by fixed requirements of

4896-451: The usability of the compress utility and other popular archivers. "gzip" is often also used to refer to the gzip file format, which is: Although its file format also allows for multiple such streams to be concatenated (gzipped files are simply decompressed concatenated as if they were originally one file), gzip is normally used to compress just single files. Compressed archives are typically created by assembling collections of files into

4968-463: The use of this standard awkward in some cases. File format identifiers are another, not widely used way to identify file formats according to their origin and their file category. It was created for the Description Explorer suite of software. It is composed of several digits of the form NNNNNNNNN-XX-YYYYYYY . The first part indicates the organization origin/maintainer (this number represents

5040-633: Was added in OpenBSD 3.4. The 'g' in this specific version stands for gratis . FreeBSD , DragonFly BSD and NetBSD use a BSD-licensed implementation instead of the GNU version; it is actually a command-line interface for zlib intended to be compatible with the GNU implementations' options. These implementations originally come from NetBSD , and support decompression of bzip2 and the Unix pack format. An alternative compression program achieving 3-8% better compression

5112-438: Was released on 1 May 1995 and was originally intended for use with the libpng image library. It is free software , distributed under the zlib License . Raw DEFLATE compressed data (RFC 1951) are typically written with a zlib or gzip wrapper encapsulating the data, by adding a header and footer. This provides stream identification and error detection that are not provided by the raw DEFLATE data. The zlib wrapper (RFC 1950)

5184-567: Was then enhanced with the Amiga standard Datatype recognition system. Another method was the FourCC method, originating in OSType on Macintosh, later adapted by Interchange File Format (IFF) and derivatives. A final way of storing the format of a file is to explicitly store information about the format in the file system, rather than within the file itself. This approach keeps the metadata separate from both

#29970