README.md
1## zlib-ng
2*zlib data compression library for the next generation systems*
3
4Maintained by Hans Kristian Rosbach
5 aka Dead2 (zlib-ng àt circlestorm dót org)
6
7|CI|Status|
8|:-|-|
9|GitHub Actions|[](https://github.com/zlib-ng/zlib-ng/actions) [](https://github.com/zlib-ng/zlib-ng/actions) [](https://github.com/zlib-ng/zlib-ng/actions)|
10|Buildkite|[](https://buildkite.com/circlestorm-productions/zlib-ng)|
11|CodeFactor|[](https://www.codefactor.io/repository/github/zlib-ng/zlib-ng)|
12|OSS-Fuzz|[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:zlib-ng)
13|Codecov|[](https://codecov.io/github/zlib-ng/zlib-ng/)|
14
15Fork Motivation
16---------------------------
17
18The motivation for this fork was due to seeing several 3rd party
19contributions containing new optimizations not getting implemented
20into the official zlib repository.
21
22Mark Adler has been maintaining zlib for a very long time, and he has
23done a great job and hopefully he will continue for a long time yet.
24The idea of zlib-ng is not to replace zlib, but to co-exist as a
25drop-in replacement with a lower threshold for code change.
26
27zlib has a long history and is incredibly portable, even supporting
28lots of systems that predate the Internet. This is great, but it does
29complicate further development and maintainability.
30The zlib code has numerous workarounds for old compilers that do not
31understand ANSI-C or to accommodate systems with limitations such as
32operating in a 16-bit environment.
33
34Many of these workarounds are only maintenance burdens, some of them
35are pretty huge code-wise. For example, the [v]s[n]printf workaround
36code has a whopping 8 different implementations just to cater to
37various old compilers. With this many workarounds cluttered throughout
38the code, new programmers with an idea/interest for zlib will need
39to take some time to figure out why all of these seemingly strange
40things are used, and how to work within those confines.
41
42So I decided to make a fork, merge all the Intel optimizations, merge
43the Cloudflare optimizations that did not conflict, plus a couple
44of other smaller patches. Then I started cleaning out workarounds,
45various dead code, all contrib and example code as there is little
46point in having those in this fork for various reasons.
47
48A lot of improvements have gone into zlib-ng since its start, and
49numerous people have contributed both small and big improvements,
50or valuable testing.
51
52Please read LICENSE.md, it is very simple and very liberal.
53
54Features
55--------
56
57* Zlib compatible API with support for dual-linking
58* Modernized native API based on zlib API for ease of porting
59* Intel deflate medium and quick algorithms
60* Support for CPU intrinsics when available
61 * Adler32 implementation using SSSE3, AVX2, Neon, & VSX
62 * Intel CRC32-B implementation using PCLMULQDQ
63 * Intel CRC32-C intrinsics for hash tables
64 * ARM CRC32-B implementation using ACLE
65 * Slide hash implementations using SSE2, AVX2, Neon, & VSX
66 * Inflate chunk copying using SSE2 & Neon
67 * Deflate hooks for IBM Z DFLTCC
68* Code sanitizers, fuzzing, and coverage
69* GitHub Actions continuous integration on Windows, macOS, and Linux
70 * Emulated CI for ARM, AARCH64, PPC, PPC64, SPARC64, S390x using qemu
71* Unaligned memory read/writes and large bit buffer improvements
72* Includes improvements from Cloudflare and Intel forks
73* Configure, CMake, and NMake build system support
74* Modern C99 syntax and a clean code layout
75* Comprehensive set of CMake unit tests
76
77Build
78-----
79
80There are two ways to build zlib-ng:
81
82### Cmake
83
84To build zlib-ng using the cross-platform makefile generator cmake.
85
86```
87cmake .
88cmake --build . --config Release
89ctest --verbose -C Release
90```
91
92Alternatively, you can use the cmake configuration GUI tool ccmake:
93
94```
95ccmake .
96```
97
98### Configure
99
100To build zlib-ng using the bash configure script:
101
102```
103./configure
104make
105make test
106```
107
108Build Options
109-------------
110| CMake | configure | Description | Default |
111|:-------------------------|:-------------------------|:--------------------------------------------------------------------------------------|---------|
112| ZLIB_COMPAT | --zlib-compat | Compile with zlib compatible API | OFF |
113| ZLIB_ENABLE_TESTS | | Build test binaries | ON |
114| WITH_GZFILEOP | --with-gzfileops | Compile with support for gzFile related functions | OFF |
115| WITH_MSAN | --with-msan | Build with memory sanitizer | OFF |
116| WITH_OPTIM | --without-optimizations | Build with optimisations | ON |
117| WITH_NEW_STRATEGIES | --without-new-strategies | Use new strategies | ON |
118| WITH_NATIVE_INSTRUCTIONS | --native | Compiles with full instruction set supported on this host (gcc/clang -march=native) | OFF |
119| WITH_SANITIZERS | --with-sanitizers | Build with address sanitizer and all supported sanitizers other than memory sanitizer | OFF |
120| WITH_FUZZERS | --with-fuzzers | Build test/fuzz | OFF |
121| WITH_MAINTAINER_WARNINGS | | Build with project maintainer warnings | OFF |
122| WITH_CODE_COVERAGE | | Enable code coverage reporting | OFF |
123
124Install
125-------
126
127WARNING: We do not recommend manually installing unless you really
128know what you are doing, because this can potentially override the system
129default zlib library, and any incompatibility or wrong configuration of
130zlib-ng can make the whole system unusable, requiring recovery or reinstall.
131If you still want a manual install, we recommend using the /opt/ path prefix.
132
133For Linux distros, an alternative way to use zlib-ng (if compiled in
134zlib-compat mode) instead of zlib, is through the use of the
135_LD_PRELOAD_ environment variable. If the program is dynamically linked
136with zlib, then zlib-ng will temporarily be used instead by the program,
137without risking system-wide instability.
138
139```
140LD_PRELOAD=/opt/zlib-ng/libz.so.1.2.11.zlib-ng /usr/bin/program
141```
142
143### Cmake
144
145To install zlib-ng system-wide using cmake:
146
147```
148cmake --build . --target install
149```
150
151### Configure
152
153To install zlib-ng system-wide using the configure script:
154
155```
156make install
157```
158
159Contributing
160------------
161
162Zlib-ng is a young project, and we aim to be open to contributions,
163and we would be delighted to receive pull requests on github.
164Just remember that any code you submit must be your own and it must
165be zlib licensed.
166Help with testing and reviewing of pull requests etc is also very
167much appreciated.
168
169If you are interested in contributing, please consider joining our
170IRC channel #zlib-ng on the Freenode IRC network.
171
172
173Acknowledgments
174----------------
175
176Thanks to Servebolt.com for sponsoring my maintainership of zlib-ng.
177
178Thanks go out to all the people and companies who have taken the time
179to contribute code reviews, testing and/or patches. Zlib-ng would not
180have been nearly as good without you.
181
182The deflate format used by zlib was defined by Phil Katz.
183The deflate and zlib specifications were written by L. Peter Deutsch.
184
185zlib was originally created by Jean-loup Gailly (compression)
186and Mark Adler (decompression).
187
188Advanced Build Options
189----------------------
190
191| CMake | configure | Description | Default |
192|:--------------------------------|:----------------------|:--------------------------------------------------------------------|------------------------|
193| ZLIB_DUAL_LINK | | Dual link tests with system zlib | OFF |
194| UNALIGNED_OK | | Allow unaligned reads | ON (x86, arm) |
195| | --force-sse2 | Assume SSE2 instructions are always available | ON (x86), OFF (x86_64) |
196| WITH_AVX2 | | Build with AVX2 intrinsics | ON |
197| WITH_SSE2 | | Build with SSE2 intrinsics | ON |
198| WITH_SSE4 | | Build with SSE4 intrinsics | ON |
199| WITH_PCLMULQDQ | | Build with PCLMULQDQ intrinsics | ON |
200| WITH_ACLE | --without-acle | Build with ACLE intrinsics | ON |
201| WITH_NEON | --without-neon | Build with NEON intrinsics | ON |
202| WITH_POWER8 | | Build with POWER8 optimisations | ON |
203| WITH_DFLTCC_DEFLATE | --with-dfltcc-deflate | Use DEFLATE COMPRESSION CALL instruction for compression on IBM Z | OFF |
204| WITH_DFLTCC_INFLATE | --with-dfltcc-inflate | Use DEFLATE COMPRESSION CALL instruction for decompression on IBM Z | OFF |
205| WITH_INFLATE_STRICT | | Build with strict inflate distance checking | OFF |
206| WITH_INFLATE_ALLOW_INVALID_DIST | | Build with zero fill for inflate invalid distances | OFF |
207| INSTALL_UTILS | | Copy minigzip and minigzip during install | OFF |
208
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}