【文章标题】:bzip3
【文章正文】: A better, faster and stronger spiritual successor to BZip2. Features higher compression ratios and better performance thanks to a order-0 context mixing entropy coder, a fast Burrows-Wheeler transform code making use of suffix arrays and a RLE with Lempel Ziv+Prediction pass based on LZ77-style string matching and PPM-style context modeling. 比BZip2更优、更快、更强的精神续作。凭借零阶上下文混合熵编码器、利用后缀数组实现的快速Burrows-Wheeler变换算法,以及基于LZ77字符串匹配和PPM上下文建模的Lempel-Ziv+Prediction游程编码,实现了更高压缩率和更优性能。
Like its ancestor, BZip3 excels at compressing text or code. 与前辈相同,BZip3尤其擅长文本和代码压缩。
If using a git clone (not needed for source packages), first…
$ ./bootstrap.sh
All…
make ./bootstrap.sh 通用编译步骤… make $ sudo make install
Alternatively, you might be able to install bzip3 using your system’s package manager: 也可通过系统包管理器安装: On macOS, you can use Homebrew to easily install: macOS用户可通过Homebrew快速安装: $ brew install bzip3
First, I have downloaded every version of Perl5 ever released and decompressed them. 首先下载所有已发布的Perl5版本并解压: % wget -r -l1 -nH —cut-dirs=2 —no-parent -A.tar.gz —no-directories https://www.cpan.org/src/5.0/ % for g in *.gz; do gunzip $g; done % ls -la | wc -l 262
Then, I put all the resulting .tar files in a single .tar file and tried to compress it using various compressors: 将所有.tar文件合并后用不同压缩器测试: xz -T16 -9 -k all.tar 10829.91s user 26.91s system 1488% cpu 14658M memory 12:09.24 total bzip2 -9 -k all.tar 981.78s user 9.77s system 95% cpu 8M memory 17:16.64 total bzip3 -e -b 256 -j 12 all.tar 2713.81s user 16.28s system 634% cpu 18301M memory 7:10.10 total bzip3 -e -b 511 -j 4 all.tar 17.65s user 12.19s system 170% cpu 12178M memory 7:08.65 total zstd -T12 -16 all.tar 4162.94s user 16.40s system 1056% cpu 687M memory 6:35.62 total
The results follow: 压缩结果如下:
| Method | Compressed size (bytes) |
|---|---|
| LZMA (xz) | 2’056’645’240 |
| bzip2 | 3’441’163’911 |
| bzip3 -b 256 | 1’001’957’587 |
| bzip3 -b 511 | 546’456’978 |
| Zstandard | 3’076’143’660 |
Finally, wall clock time decompression times (WD Blue HDD): 解压耗时测试(西数蓝盘):
| Method | Decompression time |
|---|---|
| LZMA (xz) | 4min 40s |
| bzip2 | 9min 22s |
| bzip3 (parallel) | 4min 06s |
| Zstandard | 3min 51s |
Then, I used lrzip to perform long-range deduplication on the original .tar file: 使用lrzip进行长距离去重: % time lrzip -n -o all_none.tar.lrz all.tar 546.17s user 160.87s system 102% cpu 10970M memory 11:28.00 total % time lrzip —lzma -o all_lzma.tar.lrz all.tar 702.16s user 161.87s system 122% cpu 10792M memory 11:44.83 total % time lrzip -b -o all_bzip2.tar.lrz all.tar 563.93s user 147.38s system 112% cpu 10970M memory 10:34.10 total
Finally, I compressed the resulting none.tar.lrz file using bzip3: 最终用bzip3压缩去重后文件: % time bzip3 -e -b 256 -j 2 all_none.tar.lrz 32.05s user 0.76s system 146% cpu 2751M memory 22.411 total
The results follow: 结果如下:
| Method | Compressed size (bytes) |
|---|---|
| lrzip + bzip3 | 60’672’608 |
| lrzip + lzma | 64’774’202 |
| lrzip + bzip2 | 75’685’065 |
For further benchmarks against Turbo-Range-Coder and BSC, check powturbo’s benchmark of bzip3, bzip2, bsc and others. 更多与Turbo-Range-Coder、BSC的对比测试参见powturbo的基准报告。
I TAKE NO RESPONSIBILITY FOR ANY LOSS OF DATA ARISING FROM THE USE OF THIS PROGRAM/LIBRARY, HOWSOEVER CAUSED. 本程序/库使用导致的任何数据损失,本人概不负责。
Every compression of a file implies an assumption that the compressed file can be decompressed to reproduce the original. Great efforts in design, coding and testing have been made to ensure that this program works correctly. 文件压缩的前提是假定压缩文件可被正确解压。我们已通过精心设计、编码和测试确保程序可靠性。
However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program. 但由于算法复杂性及代码中极低概率特殊案例的存在,无法完全排除潜在缺陷。
DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. 除非您能接受数据可能无法恢复的风险(即便概率极低),否则请勿使用本程序压缩关键数据。
That is not to say this program is inherently unreliable. Indeed, I very much hope the opposite is true. Bzip3/libbz3 has been carefully constructed and extensively tested. 这并非说明程序本质不可靠。相反,Bzip3/libbz3经过精心构建和充分测试。
Bzip3’s performance is heavily dependent on the compiler. x64 Linux clang13 builds usually can go as high as 17MiB/s compression and 23MiB/s decompression per thread. Windows and 32-bit builds might be considerably slower. 性能高度依赖编译器。x64 Linux clang13版本单线程压缩可达17MB/s,解压23MB/s。Windows和32位版本可能较慢。
Bzip3 has been tested on the following architectures: 已测试架构:
- x86
- x86_64
- armv6
- armv7
- aarch64
- ppc64le
- mips
- mips64
- sparc
- s390x
Check etc/BENCHMARKS.md for more results. 更多测试结果参见etc/BENCHMARKS.md。
A breakdown of components and their licenses follows: 组件及许可协议清单:
- (runtime) The codebase as a whole: Copyright 2022-2023, Kamila Szewczyk (k@iczelia.net); LGPL (LICENSE)
- (runtime) The Burrows-Wheeler transform (libsais) and LZP code: 2021-2022, Ilya Grebnov (ilya.grebnov@gmail.com); Apache 2.0 (3rdparty/libsais-LICENSE)
- (compile-time) build-aux : Copyright 2011, Daniel Richard G (skunk@iSKUNK.ORG), 2019, Marc Stevens (marc.stevens@cwi.nl), 2008, Steven G. Johnson (stevenj@alum.mit.edu); GPL-3+ with AutoConf exception
- (compile-time) build-aux/ax_check_compile_flag.m4 : Copyright 2008, Guido U. Draheim (guidod@gmx.de), 2011, Maarten Bosmans (mkbosmans@gmail.com); FSFAP
- (compile-time) build-aux/git-version-gen : Copyright 2007-2012, Free Software Foundation, Inc; GPLv3
- (runtime) bz3grep : Copyright 2003, Thomas Klausner; BSD-2-clause
bzip3 as a whole is licensed bzip3整体采用LGPL协议授权