High Efficiency Image File Format
High Efficiency Image File Format (HEIF) specifies the storage of individual images as well as image sequences in a single container file. It is an MPEG standard and defined by part 12 of MPEG-H (ISO/IEC 23008-12).
| Filename extension |
.heif, .heic |
|---|---|
| Internet media type |
image/heif, image/heic, image/heif-sequence, image/heic-sequence |
| Developed by | Moving Picture Experts Group (MPEG) |
| Type of format | Image Container Format |
| Standard | ISO/IEC CD 23008-12 |
| Website | http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format |
HEIF specification defines also the means of storing High Efficiency Video Codec (HEVC) encoded intra images and HEVC encoded image sequences in which inter prediction is applied in a constrained manner.
HEIF files are compatible with the ISO Base Media File Format (ISOBMFF, ISO/IEC 14496-12) and can also include other media streams, such as timed text and audio.
History
The requirements and main use cases of HEIF were defined in 2013.[1][2] The technical development of the specification took about 1.5 years and was finalized in summer 2015.[3]
Use Cases
Some main use cases supported by HEIF are as follows:[1][2]
- Storage of single or multiple images and their thumbnails,
- Storage of burst photos,
- Support for simultaneous capture of video and still images, i.e. storing still images and timed image sequences into the same file,
- Efficient representation of animations and cinemagraphs,
- Storage of focal and exposure stacks into the same container file,
- Storage of images derived from other images of the file, either as derived images represented by non-destructive image editing operations or as pre-computed derived images,
- Support for storing other media, such as audio and timed text, into the same container file with timed image sequences and synchronizing their playback.
As HEVC provides support for various chroma formats and sample fidelities up to lossless coding, the format can serve the whole spectrum of use cases from today's consumer devices storing images typically at 8 bits per sample to high-end professional devices with sample fidelity and dynamic range requirements going all the way up to 16 bits per sample. Some usage scenarios that are facilitated by HEIF are described below:
Digital cameras and smartphones
In order to save storage space, HEIF-encapsulated HEVC-coded images can be used for compressing the full-resolution images while keeping a lower-resolution JPEG copy (e.g. at 4K resolution or below) for on-screen displaying purposes.
Digital cameras and smartphones can use HEIF to achieve single-file packaging of burst photos, focal stacks, and exposure stacks. Similarly, simultaneously captured video and still images can be stored in the same HEIF file. HEIF also enables storage of any image collections into a single file, hence they can be shared easily.
Web pages and Internet-connected image applications
The picture element of HTML5.1 provides the capability of indicating multiple alternatives for the same image, out of which the web browser can select the one that best suits its purpose. A motivation for web pages and connected applications to start using HEIF is to reduce the web page and/or image content download times.
Image editing
Changing of the orientation and cropping are basic features of HEIF and require no re-encoding of the images. Additionally, HEIF introduces a framework for non-destructive editing operations, which can be specified by external specifications. This feature can be used by image editing applications so that the editing instructions are kept in the same file as the original image.
Features
In ISOBMFF, a continuous or timed media/metadata stream forms a track, whereas static media or metadata are stored as items. Consequently, HEIF has the following basic design:
- Still images are stored as items. All image items are independently encoded and do not depend on any other item in their decoding. Any number of image items can be included in the same file.
- Image sequences are stored as tracks. An image sequence track can be indicated to be displayed either as a timed sequence or in a non-timed manner, such as a gallery or album of images. When there is coding dependency between images, an image sequence track is used instead of image items.
- A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a HEIF file, which includes image items or image sequence tracks conforming to HEIF, and audio and/or timed text tracks conforming to any derivative format of the ISOBMFF
Keeping the above-mentioned design principles in mind, HEIF files can store the following types of data:[4]
- Image Items: storage of individual images, image properties and thumbnail(s).
- Image Derivations: derived images are generated during run-time based on descriptions such as rotation, grid and overlay. These images depend on other images stored in the HEIF file. The storage overhead of derived images is small.
- Image Sequences: storage of multiple time-related and/or temporally predicted images (like a burst-photo shot or cinemagraph animation), their properties and thumbnails. Different prediction options can be used in order to exploit the temporal and spatial similarities between the images. Hence, file sizes can be drastically reduced even when tens of images are stored in the same HEIF file.
- Auxiliary Image Items: storage of an image data which complements another image item. An alpha plane or a depth map are examples for such images. This data is not displayed as such, but used in various forms to complement another image item.
- Image Metadata: storage of EXIF, XMP and similar metadata which accompany the images stored in the HEIF file.
Examples of above-mentioned entities can be found here.
Comparison
Feature Comparison
Below table shows a comparison of different image formats:[4]
| HEIF (.heic) | JPEG/Exif | PNG | GIF (89a) | WebP | JPEG-XR / TIFF | JPEG-XR / JPX | BPG | |
|---|---|---|---|---|---|---|---|---|
| Formats and extensibility | ||||||||
| Base container file format | ISOBMFF | TIFF | - | - | RIFF | TIFF | - 4 | - |
| Lossy compression | Yes (HEVC) | Yes (JPEG) | No | No | Yes (VP8) | Yes | Yes | Yes (HEVC10) |
| Lossless compression | Yes (HEVC) | Yes (TIFF Rev 6.0) | Yes (PNG)1 | Yes (GIF)1 | Yes (VP8L) | Yes | Yes | Yes (HEVC10) |
| Extensible to other coding formats | Yes | Yes8 | No | No | No | Yes8 | Yes5 | No |
| Metadata format (on top of internal) | Exif, XMP, MPEG-7 | Exif | - | - | Exif, XMP | Exif, XMP | JPX, (XMP)6 | Exif, XMP |
| Extensible to other metadata formats | Yes | No | No | No | No | No | Yes (XML-based) | Yes |
| Other media types (audio, text, etc.) | Yes | Audio2 | No | No | No | No | Yes7 | No |
| Multi-picture features | ||||||||
| Multiple images in the same file | Yes | No11 | No | Yes3 | Yes3 | No | Yes | Yes9 |
| Image sequences / animations | Yes | No | No | Yes | Yes | No | Yes | Yes |
| Image coding | Yes | No | No | No | No | No | No | Yes |
| Derived images | ||||||||
| Multiple-of-90-degree rotations | Yes | Yes | No | No | No | Yes | Yes | No |
| Cropping | Yes | No | No | No | No | No | Yes | No |
| Tiling/overlaying | Yes | No | No | No | Yes | No | Yes | No |
| Extensible to other editing operations | Yes | No | No | No | No | No | No | No |
| Auxiliary picture information | ||||||||
| Transparency (alpha plane) | Yes | No | Yes | No12 | Yes | Yes | Yes | Yes |
| Depth map | Yes | No | No | No | No | No | No | No |
| Thumbnail image | Yes | Yes | No | No | No | Yes | Yes | Yes |
1 In GIF and indexed color PNG encoding, lossy color quantization is applied while the color-quantized image is losslessly compressed.
2 PCM, µ-Law PCM and ADPCM encapsulated in RIFF WAV
3 Only for animations and tiling/overlaying
4 JPX is a box-structured format compatible with ISOBMFF. However, only the File Type box is common in JPX and ISOBMFF.
5 Encapsulation of JPEG-2000 and JPEG-XR have been specified for JPX container. Mappings for other codecs could be similarly specified.
6 JPX (ITU-T T.800 and T.801) specifies an own metadata schema, but is capable of carrying an XML formatted metadata, such as XMP.
7 JPX can contain media complying with ISOBMFF (or derivatives thereof). No accurate synchronization between JPX animations and other media.
8 TIFF as a container format facilitates extensions to other coding formats.
9 Only for animations, thumbnails, and alpha planes. Non-timed image collections not supported.
10 HEVC Main 4:4:4 16 Still Picture profile, Level 8.5, with additional constraints
11 Can be enabled through the MP extension
12 A palette index for fully transparency can be specified.
The following references are used in the comparison:
| Image format | Version or date | Reference and/or URL |
|---|---|---|
| HEIF (.heic) | 03/2015 | ISO/IEC 23008-12 |
| JPEG | ISO/IEC 10918-1 | ITU-T Rec. T.81 | |
| Exif | http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf | |
| PNG | http://www.w3.org/TR/PNG/ | |
| GIF | 89a | http://www.w3.org/Graphics/GIF/spec-gif89a.txt |
| WebP | https://developers.google.com/speed/webp/docs/riff_containerVP8L: https://developers.google.com/speed/webp/docs/webp_lossless_bitstream_specification | |
| JPEG-XR | ISO/IEC 29199-2 | ITU-T Rec. T.832
ISO/IEC 15444-2 | ITU-T Rec. T.801 (for JPX) | |
| BPG | 0.9.5 | http://bellard.org/bpg/bpg_spec.txt |
Note: It is acknowledged that a summary such as that in the comparison table might be somewhat incomplete when it comes to features of different formats. For example, the table does not cover some of the extensions of JPEG.
Compression performance comparison
The table below illustrates coding efficiency of HEVC intra coding with respect to well-known still picture codecs. The results indicate that JPEG would require 2.39 times the file size in order to achieve the same objective picture quality. For JPEG-XR this number is 1.66 and for JPEG-2000 it is 1.44.
| HEVC intra coding performance with respect to legacy
formats. File size multiplier factors which represent the compression ratios against the listed picture codecs[4] | |||||
|---|---|---|---|---|---|
|
Class |
Resolution |
Characteristics |
JPEG |
JPEG XR |
JPEG 2000 |
| Class A | 2560x1600 | Cropped 4Kx2K sequences for Ultra HDTV services | 1.87 | 1.44 | 1.48 |
|
Class B |
1920x1080 |
High resolution sequences for streaming and broadcast services |
2.24 |
1.62 |
1.15 |
|
Class C |
832x480 |
Medium resolution sequences for internet/mobile video services |
2.22 |
1.53 |
1.50 |
|
Class C |
832x480 |
Medium resolution sequences for internet/mobile video services |
2.22 |
1.53 |
1.50 |
|
Class D |
416x240 |
Low resolution sequences for services to resource constrained devices |
2.10 |
1.47 |
1.43 |
|
Class E |
1280x720 |
720p sequences for video conferencing applications |
2.70 |
1.73 |
1.23 |
|
Class F |
1024x768, 1280x720 |
Computer screen content and computer generated content |
3.23 |
2.18 |
1.87 |
|
Average |
2.39 |
1.66 |
1.44 | ||
For natural content, the restricted inter coding can typically provide two to three times better compression than intra picture coding. In special cases like animations where majority of the scene is static the compression efficiency can significantly exceed those levels and be tens of times more efficient than intra coding. the following table provides several comparison information:
| Coding efficiency improvements provided by low latency predictive coding of the HEVC Image File Format. Bitrate impact and coding gain are reported with respect to HEVC intra coding[4] | ||||
|---|---|---|---|---|
|
Content |
Type |
Frames |
Bitrate change |
Coding gain |
| Class A | Image burst | 8 | -46% | 1.9 |
|
Class B |
Image burst |
8 | -51% | 2.0 |
|
Class C |
Image burst |
8 | -60% | 2.5 |
|
Class D |
Image burst |
8 | -63% | 2.7 |
|
Class E |
Image burst |
8 | -79% | 4.8 |
|
Class F |
Image burst |
8 | -55% | 2.2 |
|
Memorial |
Exposure stack |
16 | -29% | 1.4 |
|
Mersu |
Focal stack |
13 | -25% | 1.3 |
|
Car&Tractor |
Cinemagraph |
48 | -97% | 33.3 |
Syntax and Structure
HEIF conforms to the ISO Base Media File Format (ISO/IEC 14496-12). Image items are stored in root level meta box and image sequences are stored as video media tracks.
A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a file, which includes image items or image sequence tracks conforming to HEIF, and audio and timed text tracks conforming to any derivative format of the ISOBMFF.
Files conforming to ISOBMFF consist of a sequence of data structures called boxes, each containing a four-character code (4CC) indicating the type of the box, the size of the box in terms of bytes, and the payload of the box. Boxes may be nested, i.e. a box may contain other boxes. ISOBMFF and HEIF specify constraints on the allowed box order and hierarchy.
Image Items
The roles of image items as specified in HEIF are listed and described below. Note that a single image can be associated with more than one role.
|
Role |
Description |
|---|---|
|
coded image |
A coded representation of an image. |
|
derived image |
An image that is represented in a file by an indicated operation to indicated input images and can be obtained by performing the operation to the input images. |
|
cover image (a.k.a. primary item) |
A representative image of the image items and image sequence tracks of the file. The cover image should be displayed when no other information is available on the preference to display the image items of the file. The file can have only one cover image. |
|
thumbnail image |
A smaller-resolution representation of a master image. A master image can have multiple thumbnail images (e.g. different resolutions). |
|
auxiliary image |
An image that complements a master image. For example, an alpha plane or a depth map. Can assist in displaying the master image but is not typically displayed as such. |
|
master image |
An image that is not a thumbnail image or an auxiliary image. It typically represents a full-resolution displayable image. |
|
hidden image |
An image that should never be displayed. Can be present in the file for example as an input image for a derived image. |
|
pre-derived coded image |
A coded image that has been derived from other images. For example, a high dynamic range image derived from an exposure-bracketed set of images. The input images used for derivation are linked to the final pre-derived coded image by referencing. No derivation operation is defined. |
HEIF allows the storage of image properties which are shared among different image items. There are two types of properties: descriptive and transformative. Descriptive properties provide information about the image item without applying modifications on the image itself. Transformative properties provide information about the transformative modification that needs to be done on the image item. The HEIF standard specifies transformative properties for rotation by 90, 180, and 270 degrees and cropping. In addition to descriptive image properties, image items can optionally be characterized with metadata items, the format of which follows e.g. Exif or XMP metadata.
Derived Images
Derived images enable non-destructive image editing, where the original coded images are kept in the file, while new images, called derived images, can be introduced by specifying a transformation operation that is applied to one or more input images. HEIF specifies the generic structures used for storing derived images as items as well as a few specific types of derived images. The item type of a derived image item indicates the transformation operation, while the item payload contains the input parameters to the operation. The input images can be coded images or derived images. The derived image types specified in the HEIF standard are listed below. Other types may be specified in other documents or later versions of the HEIF standard.
| Derived Image Items | ||
|---|---|---|
|
Name |
Item 4CC |
Description |
|
Identity transformation |
iden |
Cropping and/or rotation by 90, 180, or 270 degrees, imposed through the respective transformative properties. |
|
Image overlay |
iovl |
Overlaying any number of input images in indicated order and locations onto the canvas of the output image. |
|
Image grid |
grid |
Reconstructing a grid of input images of the same width and height. |
Image Sequences
An image sequence track inherits the properties and features of a video track of ISOBMFF. A key difference between an image sequence track and a video track lies in the interpretation of the timing information given for the track: While the timing information given for a video track adheres the decoding and output timing of the contained video bitstream, the timing information of an image sequence track can represent the capture times (e.g. for separate images of an exposure stack) or the suggested display timing (e.g. for a slide show). It can be indicated whether an image sequence track should be played as a timed sequence or by some non-timed means, such as an image gallery.
HEIF allows the use of inter-picture prediction (a.k.a. inter coding), which can provide significant coding efficiency improvement to image sequences when images are correlated. Such correlation is especially evident in content like image bursts or animation clips. When inter coding is in use, accessing and decoding a particular picture from an image sequence also requires decoding of its reference pictures for inter prediction. HEIF specifies indications assisting players in random access.
Roles can be indicated for image sequence tracks similarly to image items. Below is the list of roles of the image sequence tracks:
| Roles of image sequence tracks | |
|---|---|
|
Role |
Description |
|
thumbnail image sequence |
A smaller-resolution representation of a master image sequence. |
|
auxiliary image sequence |
An image sequence that complements a master image sequence. For example, a sequence of alpha plane or depth map images. Can assist in displaying the master image sequence but is not typically displayed as such. |
|
master image sequence |
An image sequence that is not a thumbnail image sequence or an auxiliary image sequence. Typically contains full-resolution displayable images. |
ISOBMFF specifies indications for video and image sequence tracks, such as width and height of the coded images, color information, such as the ICC profile or color primaries, rectangular cropping, transformation matrix, e.g. for rotation. Additionally, timed metadata track(s) can be used to convey Exif, XMP, and MPEG-7 metadata applying to time-parallel samples of the linked image sequence track.
The most important features that enable controlling the playback of an HEIF file are listed below. Some of these features were introduced in the ISOBMFF or ISO/IEC 14496-15 and are explicitly inherited by HEIF, while other features were specifically designed for the HEIF standard.
| Features controlling Image sequence playback | ||
|---|---|---|
|
Feature |
First appeared in |
Description |
| non-displayable sample | ISO/IEC 14496-15 | Is never displayed, but can be used as a reference for predicting other images in the track. |
|
timed vs. non-timed playback |
HEIF |
In timed playback, the image sequence is played as video, whereas in non-timed playback the samples of the track are displayed by other means, such as an image gallery. Non-timed playback may be indicated e.g. when a track is used for achieving a better compression efficiency for an exposure stack. |
|
edit list |
ISOBMFF |
A list of ranges of the image sequence track in their playback order. Enables modifying the playback order and pace of samples. |
|
looping |
HEIF |
HEIF allows indicating edit list repetition e.g. for looping animations. The repetition can be indicated to last for a certain duration or be infinite. |
|
cropping and rotation |
ISOBMFF |
Rectangular cropping and rotation by 90, 180, 270 degrees can be specified. |
Supported Media Codecs
HEIF specifies a structural format, from which codec-specific image formats can be derived. HEIF also includes the specification for encapsulating images and image sequences conforming to the High Efficiency Video Coding (HEVC, ISO/IEC 23008-2 | ITU-T Rec. H.265). Future versions of the standard may include other video and still image media codecs.
HEVC Image File Format
The HEIF standard includes the specification of encapsulating HEVC-coded images and image sequences into HEIF-compliant files. The specification includes the following aspects:
- The sample entry format and the sample format of HEVC image sequence tracks are identical to those of the HEVC video tracks specified in ISO/IEC 14496-15. For compatibility with HEVC-capable video players, it is therefore advisable to create both an HEVC image sequence track and an HEVC video track that refer to the same coded images.
- The format of the decoder configuration information in the image entry item is identical to that in the sample entry format. Likewise, the item format is identical to the sample format of HEVC video and image sequence tracks. This enables having the same intra-coded image as an image item and as a sample in an HEVC image sequence or video track.
- The configuration information of alpha planes and depth maps follows the respective supplemental enhancement information of HEVC.
- HEVC image players are required to support rectangular cropping and rotation by 90, 180, and 270 degrees. The primary use case for the mandatory support for rotation by 90 degrees is for the photo shooting situations in which the camera orientation is incorrectly detected or concluded. This requirement makes it possible to manually adjust the image or image sequence orientation afterwards without the need for re-encoding the image or image sequence. Similarly, cropping may be useful to enable post-shooting zoom without the need for re-encoding. As rotation by 90, 180, or 270 degrees as well as cropping are mandatory for all HEVC image file players, it is guaranteed that re-encoding is not required to carry out these operations.
- Samples in image sequence tracks must be either intra-coded images or inter-picture predicted images with reference to only intra-coded images. These constraints of inter-picture prediction reduce the decoding latency for accessing any particular image within an HEVC image sequence track.
Brands and MIME Types
Files conforming to HEIF start with a FileTypeBox as defined in the ISOBMFF standard, which contains a list of brands the file complies with. As the FileTypeBox is located at the start of the file, it provides easily accessible indications of the file contents to media players. Each brand is identified by its unique four-character code. The specification of a brand can include requirements and constraints for files of the brand and for file players supporting the brand. A brand included in the FileTypeBox permits a player that supports the requirements of the brand to play the file.
The brands specified in the HEIF standard are presented in the table below. The HEIF standard specifies the 'mif1' and 'msf1' structural brands. Additionally, HEVC-specific brands are specified as listed in Table I. The dedicated brand names, 'heic' and 'hevc' indicates that HEVC Main Profile is utilized.
| Brands, MIME subtypes, and file extensions for HEIF | |||||
|---|---|---|---|---|---|
| Brand | Coding format | Image or sequence? | MIME Type | MIME subtype | File extension |
| mif1 | Any | image | image | heif | .heif |
| msf1 | Any | sequence | image | heif-sequence | .heif |
| heic | HEVC (Main or Main Still Picture profile) | image | image | heic | .heic |
| heix | HEVC (Main 10 or format range extensions profile) | image | image | heic | .heic |
| hevc | HEVC (Main or Main Still Picture profile) | sequence | image | heic-sequence | .heic |
| hevx | HEVC (Main 10 or format range extensions profile) | sequence | image | heic-sequence | .heic |
Internet media types, also known as MIME (Multipurpose Internet Mail Extensions) types are used by various applications to identify the type of a resource or a file. MIME types consist of a media type ('image' in the case of HEIF files), a subtype, and zero or more optional parameters. For multi-purpose files, the selection of the subtype can be made on the basis of the primary use of the file.
An optional codecs MIME parameter can be present to indicate the used coding formats of the tracks and items present in the file. The codecs MIME parameter also includes the profile-tier-level value to which an HEVC-coded image item or an image sequence track conforms.
As of January 2016 image/heif, image/heif-sequemce, and image/heic were not registered as MIME types.[5]
Related Standards
- MPEG-H
- HEVC (H.265)
- ISO Base Media File Format
Other Image Formats
Implementations
- High Efficiency Image File Format web-site and publicly available source code (Nokia Technologies, 2015): The source code is written in C++. The reader implementation and API is also ported to JavaScript and used in the HEIF web-site. The source code repository contains the following components:
- ISO Base Media File Format (ISOBMFF) parser/writer source code
- HEIF Reader API and Library
- HEIF Writer Library and Example Application
- HEIF Reader API Usage Example Code
- HEIF Source Code Documentation
- HEIF Web Site Content
- HEIF Reader JavaScript Implementation
References
- 1 2 "Requirements for still image coding using HEVC | MPEG". mpeg.chiariglione.org. Retrieved 2015-11-25.
- 1 2 "Requirements for HEVC image sequences | MPEG". mpeg.chiariglione.org. Retrieved 2015-11-25.
- ↑ "Evidence motivates MPEG to launch new standardization effort for HDR".
- 1 2 3 4 5 M. M. Hannuksela, E. B. Aksu, V. K. Malamal Vadakital, J. Lainema. "Overview of the High Efficiency Image File Format". JCT-VC document JCTVC-V0072, Oct. 2015.
- ↑ "Media Types". IANA. 2016-01-14. Retrieved 2016-01-18.
External links
- MPEG Image File Format standard web-site
- High Efficiency Image File Format web-site
- High Efficiency Image File Format Source Code at GitHub