README.md
1Zstandard library files
2================================
3
4The __lib__ directory is split into several sub-directories,
5in order to make it easier to select or exclude features.
6
7
8#### Building
9
10`Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions),
11including commands variables, staged install, directory variables and standard targets.
12- `make` : generates both static and dynamic libraries
13- `make install` : install libraries and headers in target system directories
14
15`libzstd` default scope is pretty large, including compression, decompression, dictionary builder,
16and support for decoding legacy formats >= v0.5.0.
17The scope can be reduced on demand (see paragraph _modular build_).
18
19
20#### Multithreading support
21
22Multithreading is disabled by default when building with `make`.
23Enabling multithreading requires 2 conditions :
24- set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`)
25- for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`)
26
27Both conditions are automatically applied when invoking `make lib-mt` target.
28
29When linking a POSIX program with a multithreaded version of `libzstd`,
30note that it's necessary to invoke the `-pthread` flag during link stage.
31
32Multithreading capabilities are exposed
33via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351).
34
35
36#### API
37
38Zstandard's stable API is exposed within [lib/zstd.h](zstd.h).
39
40
41#### Advanced API
42
43Optional advanced features are exposed via :
44
45- `lib/common/zstd_errors.h` : translates `size_t` function results
46 into a `ZSTD_ErrorCode`, for accurate error handling.
47
48- `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`,
49 it unlocks access to the experimental API,
50 exposed in the second part of `zstd.h`.
51 All definitions in the experimental APIs are unstable,
52 they may still change in the future, or even be removed.
53 As a consequence, experimental definitions shall ___never be used with dynamic library___ !
54 Only static linking is allowed.
55
56
57#### Modular build
58
59It's possible to compile only a limited set of features within `libzstd`.
60The file structure is designed to make this selection manually achievable for any build system :
61
62- Directory `lib/common` is always required, for all variants.
63
64- Compression source code lies in `lib/compress`
65
66- Decompression source code lies in `lib/decompress`
67
68- It's possible to include only `compress` or only `decompress`, they don't depend on each other.
69
70- `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples.
71 The API is exposed in `lib/dictBuilder/zdict.h`.
72 This module depends on both `lib/common` and `lib/compress` .
73
74- `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`.
75 This module depends on `lib/common` and `lib/decompress`.
76 To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation.
77 Specifying a number limits versions supported to that version onward.
78 For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0".
79 Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats".
80 By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`.
81 Decoding supported legacy format is a transparent capability triggered within decompression functions.
82 It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`.
83 Each version does also provide its own set of advanced API.
84 For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` .
85
86- While invoking `make libzstd`, it's possible to define build macros
87 `ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`,
88 and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the
89 corresponding features. This will also disable compilation of all
90 dependencies (eg. `ZSTD_LIB_COMPRESSION=0` will also disable
91 dictBuilder).
92
93- There are a number of options that can help minimize the binary size of
94 `libzstd`.
95
96 The first step is to select the components needed (using the above-described
97 `ZSTD_LIB_COMPRESSION` etc.).
98
99 The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This
100 disables various optional components and changes the compilation flags to
101 prioritize space-saving.
102
103 Detailed options: Zstandard's code and build environment is set up by default
104 to optimize above all else for performance. In pursuit of this goal, Zstandard
105 makes significant trade-offs in code size. For example, Zstandard often has
106 more than one implementation of a particular component, with each
107 implementation optimized for different scenarios. For example, the Huffman
108 decoder has complementary implementations that decode the stream one symbol at
109 a time or two symbols at a time. Zstd normally includes both (and dispatches
110 between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or
111 `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding
112 compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT`
113 and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of
114 only one or the other of two decompression implementations. The smallest
115 binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and
116 `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`).
117
118 For squeezing the last ounce of size out, you can also define
119 `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`,
120 which removes the error messages that are otherwise returned by
121 `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`).
122
123 Finally, when integrating into your application, make sure you're doing link-
124 time optimation and unused symbol garbage collection (via some combination of,
125 e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`,
126 `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`,
127 `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands
128 the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your
129 compiler's documentation.
130
131- While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1`
132 will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in
133 the shared library, which is now hidden by default.
134
135- The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries
136 which can detect at runtime the presence of BMI2 instructions, and use them only if present.
137 These instructions contribute to better performance, notably on the decoder side.
138 By default, this feature is automatically enabled on detecting
139 the right instruction set (x64) and compiler (clang or gcc >= 5).
140 It's obviously disabled for different cpus,
141 or when BMI2 instruction set is _required_ by the compiler command line
142 (in this case, only the BMI2 code path is generated).
143 Setting this macro will either force to generate the BMI2 dispatcher (1)
144 or prevent it (0). It overrides automatic detection.
145
146- The build macro `ZSTD_NO_UNUSED_FUNCTIONS` can be defined to hide the definitions of functions
147 that zstd does not use. Not all unused functions are hidden, but they can be if needed.
148 Currently, this macro will hide function definitions in FSE and HUF that use an excessive
149 amount of stack space.
150
151- The build macro `ZSTD_NO_INTRINSICS` can be defined to disable all explicit intrinsics.
152 Compiler builtins are still used.
153
154
155#### Windows : using MinGW+MSYS to create DLL
156
157DLL can be created using MinGW+MSYS with the `make libzstd` command.
158This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`.
159The import library is only required with Visual C++.
160The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to
161compile a project using gcc/MinGW.
162The dynamic library has to be added to linking options.
163It means that if a project that uses ZSTD consists of a single `test-dll.c`
164file it should be linked with `dll\libzstd.dll`. For example:
165```
166 gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll
167```
168The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
169
170
171#### Advanced Build options
172
173The build system requires a hash function in order to
174separate object files created with different compilation flags.
175By default, it tries to use `md5sum` or equivalent.
176The hash function can be manually switched by setting the `HASH` variable.
177For example : `make HASH=xxhsum`
178The hash function needs to generate at least 64-bit using hexadecimal format.
179When no hash function is found,
180the Makefile just generates all object files into the same default directory,
181irrespective of compilation flags.
182This functionality only matters if `libzstd` is compiled multiple times
183with different build flags.
184
185The build directory, where object files are stored
186can also be manually controlled using variable `BUILD_DIR`,
187for example `make BUILD_DIR=objectDir/v1`.
188In which case, the hash function doesn't matter.
189
190
191#### Deprecated API
192
193Obsolete API on their way out are stored in directory `lib/deprecated`.
194At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`.
195These prototypes will be removed in some future version.
196Consider migrating code towards supported streaming API exposed in `zstd.h`.
197
198
199#### Miscellaneous
200
201The other files are not source code. There are :
202
203 - `BUCK` : support for `buck` build system (https://buckbuild.com/)
204 - `Makefile` : `make` script to build and install zstd library (static and dynamic)
205 - `README.md` : this file
206 - `dll/` : resources directory for Windows compilation
207 - `libzstd.pc.in` : script for `pkg-config` (used in `make install`)
208