588 Commits

Author	SHA1	Message	Date
Yann Collet	0d58aaf6f0	/contrib: fixed license header ... removed last reference to PATENTS file	2017-10-02 02:07:17 -07:00
Yann Collet	f1a3ce34b7	Merge pull request #826 from terrelln/license ... [linux-kernel] Update license	2017-08-31 12:52:23 -07:00
Nick Terrell	f9252d8347	[linux-kernel] Update license	2017-08-31 12:48:36 -07:00
Yann Collet	b0cb081dc8	last batch of header files changed to reflect new license (#825) ... only remains to update contrib/linux-kernel (@terrelln)	2017-08-31 12:20:50 -07:00
Yann Collet	e21384fffb	fixed more file headers after license change (#825)	2017-08-31 12:11:57 -07:00
Yann Collet	e9dc204f42	fixed a bunch of headers after license change (#825)	2017-08-31 11:24:54 -07:00
Yann Collet	394bdd7db9	changed license for examples ... intentionnally this time	2017-08-29 09:24:11 -07:00
Yann Collet	dd63533b29	Merge pull request #797 from paulcruz74/documentation ... Updating README.md	2017-08-16 00:37:55 -07:00
Paul Cruz	733ca51360	Updating README.md	2017-08-15 17:48:23 -07:00
Yann Collet	2daeab12f4	Merge pull request #795 from terrelln/squashfs-tools2 ... [kernel] Update squashfs-tools patch	2017-08-15 11:59:41 -07:00
Nick Terrell	57e2df6651	[kernel] Update squashfs-tools patch	2017-08-14 22:43:36 -07:00
Paul Cruz	fc90469587	updated program name print statement	2017-08-10 16:11:59 -07:00
Paul Cruz	75c3ad947c	Merge remote-tracking branch 'upstream/dev' into dev	2017-08-10 14:46:31 -07:00
Yann Collet	8915b9b518	Merge pull request #786 from terrelln/squashfs-tools ... [linux-kernel] Update patches for v4 and v5	2017-08-09 16:00:11 -07:00
Nick Terrell	8b6702a00d	[linux-kernel] Update patches for v5	2017-08-09 13:03:40 -07:00
Yann Collet	a1fe656165	Merge pull request #779 from paulcruz74/adapt-approach-4 ... Patching warnings, adding ability to bound compression variation	2017-08-07 14:50:38 -07:00
Paul Cruz	7069bb9458	Merge branch 'adapt-approach-4' into bug-fixes	2017-08-07 13:18:53 -07:00
Paul Cruz	e100a311eb	removed direct assignment of 22, used ZSTD_maxCLevel() instead	2017-08-07 13:11:07 -07:00
Nick Terrell	7393b49fbd	[linux-kernel] Update patches for v4	2017-08-04 16:57:03 -07:00
Paul Cruz	01237e3b35	changed multi to zstd-adaptive in the help menu	2017-08-03 15:13:49 -07:00
Paul Cruz	8be7bba08c	added mutex for compression level to avoid data race	2017-08-02 10:27:33 -07:00
Paul Cruz	69ef22c0ac	added detach statements to prevent resource leak	2017-08-01 17:36:13 -07:00
Paul Cruz	f8c7b191e7	Merge branch 'adapt-approach-4' into dev	2017-07-31 15:42:57 -07:00
Paul Cruz	0295737ad7	change signal to broadcast for jobCompressed condition varaible since multiple threads waiting	2017-07-31 13:43:03 -07:00
Paul Cruz	cf92221671	Merge remote-tracking branch 'upstream/dev' into dev	2017-07-31 11:43:40 -07:00
Paul Cruz	c2f1bda8a1	Merge branch 'adapt-approach-4' into dev	2017-07-31 11:43:01 -07:00
Paul Cruz	9ea7df03de	add install target in makefile	2017-07-31 11:04:17 -07:00
Paul Cruz	f60cd3f99b	print defaults and range, remove EXT	2017-07-31 09:47:09 -07:00
Paul Cruz	e22b60cb76	removed ternary operation, added assert statement, check to make sure initial compression level is within bounds	2017-07-28 17:46:51 -07:00
Paul Cruz	cb9af53e77	delete empty line	2017-07-28 17:28:25 -07:00
Paul Cruz	51788225db	remove exe extension from makefile, reinclude pthread flag	2017-07-28 17:27:36 -07:00
Yann Collet	ee27f6937c	Merge pull request #758 from stellamplau/ldm ... Add working prototype of a long distance matcher	2017-07-28 17:07:21 -07:00
Paul Cruz	4d904ac800	add flags for multithreading	2017-07-28 16:12:58 -07:00
Paul Cruz	0f4cb67b00	add tests for compression bounds, fix another warning	2017-07-28 15:55:02 -07:00
Paul Cruz	ff54fced64	patched style errors, add ability to bound compression level variation	2017-07-28 15:30:46 -07:00
Stella Lau	8fae41c412	Return error code in verify() and minor code cleanup	2017-07-27 17:14:05 -07:00
Stella Lau	1294a4a897	Fix typo	2017-07-27 15:49:46 -07:00
Stella Lau	627621839c	Add checks in initialization code	2017-07-27 15:37:37 -07:00
Stella Lau	c105f605e6	Update README	2017-07-27 11:11:35 -07:00
Yann Collet	e1222544be	Merge pull request #753 from paulcruz74/adapt-approach-3 ... adaptive compression v1	2017-07-27 10:00:10 -07:00
Paul Cruz	2320e7378a	remove unused variable, add documentation for context fields	2017-07-26 17:02:47 -07:00
Stella Lau	9eaf3d22d0	Allow HASH_ONLY_EVERY_LOG to be configured in ldm.h	2017-07-26 16:43:25 -07:00
Paul Cruz	ab5a78547e	fix leaky abstraction regarding measuring completion	2017-07-26 16:40:05 -07:00
Paul Cruz	715f36ca81	added definitions for conversion constants, moved forced compression check to top of adaptCompressionLevel, used ZSTD_BLOCKSIZE_MAX	2017-07-26 15:52:15 -07:00
Paul Cruz	6c1c1242fc	set the window log value before performing compression	2017-07-26 14:29:59 -07:00
Stella Lau	40759bade9	Add README and clean up code	2017-07-26 13:24:03 -07:00
Paul Cruz	a959cc881a	moved reset of completion to right after wait	2017-07-26 10:34:48 -07:00
Paul Cruz	305d5ee70f	change to >= convergence counter	2017-07-26 10:20:29 -07:00
Paul Cruz	be92a38d6a	decrease completion requirements for change, move create thread wait, merge cases where compression thread should wait	2017-07-26 10:05:10 -07:00
Stella Lau	e9161637b2	Allow parameters to be modified from a separate file	2017-07-25 18:13:27 -07:00
Paul Cruz	0b18d21e03	building on readme, added another help tip in the menu	2017-07-25 17:47:02 -07:00
Paul Cruz	7cc74e0b27	adding more to readme	2017-07-25 16:55:16 -07:00
Paul Cruz	8dbb07d822	updated progress bar with better representation of time, added const	2017-07-25 16:03:43 -07:00
Paul Cruz	9a132707af	changing time units to seconds	2017-07-25 15:26:26 -07:00
Stella Lau	629c300118	Rename and remove unneeded files	2017-07-25 15:17:36 -07:00
Paul Cruz	31a9ed9883	updated const values, added more comments	2017-07-25 14:53:40 -07:00
Paul Cruz	5cfbf609a4	removed old debug statements no longer being used	2017-07-25 14:31:48 -07:00
Paul Cruz	0882cd1981	progress bar -- don't print num jobs, time elapsed shown in seconds	2017-07-25 14:26:55 -07:00
Paul Cruz	310c12d07e	moved debug statements to a compiler flag	2017-07-25 14:08:39 -07:00
Paul Cruz	e02c79f833	started using decrease cooldown so that compression level would not decrease several times in a row	2017-07-25 11:16:27 -07:00
Paul Cruz	85d7c919f6	created independent function for controlling how completion relates to compression level change	2017-07-25 10:32:14 -07:00
Paul Cruz	6f1e260edd	added mechanism for getting rid of spikes	2017-07-25 10:01:10 -07:00
Paul Cruz	700758d676	added help statement for -p, switched it to hide progress bar now that progress bar is default	2017-07-24 16:26:20 -07:00
Paul Cruz	df3754b6ed	add quiet option, make progress bar default	2017-07-24 16:19:07 -07:00
Stella Lau	0295a27133	Experiment with not using a checksum	2017-07-24 15:26:44 -07:00
Paul Cruz	4dc83ca64c	compression thread should take measurements independently based on whether or not the create/write thread will actually bottleneck performance	2017-07-24 15:14:58 -07:00
Paul Cruz	0ee3f8c2f8	adding more debug	2017-07-24 15:06:11 -07:00
Paul Cruz	8328f8192a	updating debug statements again	2017-07-24 14:40:23 -07:00
Paul Cruz	d3d759301f	changing position of endline for debug	2017-07-24 13:47:39 -07:00
Stella Lau	08a6e9a141	Minor code cleanup	2017-07-24 13:22:00 -07:00
Stella Lau	6eefa32911	Deduplicate code	2017-07-24 12:40:59 -07:00
Stella Lau	8ed9220102	Experiment with eviction policies and minor code cleanup	2017-07-24 12:05:43 -07:00
Paul Cruz	e508f632d6	updated comments and debug statements	2017-07-24 11:01:36 -07:00
Stella Lau	eb16da647d	Minor clean up	2017-07-24 10:20:53 -07:00
Stella Lau	1a188fe864	Fix overflow bug when calculating hash	2017-07-24 10:20:53 -07:00
Stella Lau	0b8fb1703b	Experiment with 64-bit hash insertion policy	2017-07-24 10:20:30 -07:00
Stella Lau	273c17b350	Experiment with 64-bit hash and checksum	2017-07-24 10:19:50 -07:00
Paul Cruz	483d936b87	reduced competition for completion mutex by separating mutex use based on which values is updated	2017-07-23 14:09:16 -07:00
Paul Cruz	880f08d104	change how completion is measured in compression thread	2017-07-23 10:18:54 -07:00
Paul Cruz	08d9e42ec6	removed useless measurements	2017-07-21 18:02:55 -07:00
Paul Cruz	95bef759b3	switched over to model where reading only waits on compression thread	2017-07-21 17:49:39 -07:00
Paul Cruz	6455ec482c	taking the maximum of the completion level reads in order to determine which one was waiting more	2017-07-21 16:05:01 -07:00
Paul Cruz	05fe8dd47c	updating debug statements	2017-07-21 14:06:24 -07:00
Paul Cruz	db109f8fef	measure multiple completion levels during each wait	2017-07-21 13:38:24 -07:00
Paul Cruz	721c6a8b97	added bounding to compression level change	2017-07-21 09:30:24 -07:00
Paul Cruz	e929d3b787	added priority decision making for adapt compression level	2017-07-21 09:26:35 -07:00
Paul Cruz	9259c7afa4	semi working version that stabilizes	2017-07-20 18:45:33 -07:00
Paul Cruz	82e488770c	fixed bug where writeSize could be zero	2017-07-20 16:38:02 -07:00
Paul Cruz	a19916425d	reworked adaptCompressionLevel to only account for completion information	2017-07-20 16:19:16 -07:00
Nick Terrell	7d3ac0710d	[linux] Update patches for v3	2017-07-20 13:33:55 -07:00
Paul Cruz	7ab758a640	changed how completion is actually sampled	2017-07-20 10:53:51 -07:00
Stella Lau	13a01ffb27	Fix off-by-one in size calculations	2017-07-19 17:24:09 -07:00
Stella Lau	2427a154cb	Minor refactoring	2017-07-19 16:56:28 -07:00
Paul Cruz	dcf609f835	make adaptCompressionLevel oscillate less	2017-07-19 16:36:33 -07:00
Paul Cruz	2a22c7915e	call ZSTD_compressBegin() once	2017-07-19 16:00:54 -07:00
Paul Cruz	6767abe652	fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0)	2017-07-19 14:54:15 -07:00
Stella Lau	030264ca51	Experiment with integrating ZSTD_count with findBestMatch	2017-07-19 14:14:26 -07:00
Paul Cruz	42382c1216	added some debug statements, adjusted end condition	2017-07-19 13:30:07 -07:00
Paul Cruz	5a85c57e30	set up new calculations compression completion progress	2017-07-19 11:47:17 -07:00
Paul Cruz	f1ac518b59	split compression into smaller blocks	2017-07-19 11:23:40 -07:00
Paul Cruz	338951cd48	moved compression adapt to avoid warning	2017-07-19 10:23:46 -07:00
Paul Cruz	4497ecf297	change compression level only right before actually performing compression. When waiting, only update waiting statistics.	2017-07-19 10:14:00 -07:00
Paul Cruz	e11bf55d0b	added mechanism for measuring how much of a job has been created	2017-07-19 10:10:47 -07:00
Paul Cruz	559ea4ff25	split up read process into smaller chunks	2017-07-19 09:59:17 -07:00
Paul Cruz	6119cd2164	added additional print for help menu	2017-07-19 09:43:17 -07:00
Stella Lau	4352e09cb0	Avoid recounting match lengths with ZSTD_count	2017-07-18 18:35:25 -07:00
Stella Lau	1fa223859f	Switch to using ZSTD_count instead of function pointer	2017-07-18 18:05:10 -07:00
Paul Cruz	3d7f1afadd	changed createCCtx() to split into initialization and creation	2017-07-18 17:32:36 -07:00
Paul Cruz	2c4e4ddc50	added mutex for stats struct	2017-07-18 15:55:58 -07:00
Paul Cruz	ad66faf16a	added progress check for filewriting, put important shared data behind mutex when being read from/written to	2017-07-18 15:23:11 -07:00
Stella Lau	19258f51c1	Make the meaning of LDM_MEMORY_USAGE consistent across tables	2017-07-18 14:25:39 -07:00
Paul Cruz	a34bc30237	setting up basic readme	2017-07-18 13:31:02 -07:00
Paul Cruz	29c36cf051	rename completion variable, split up fwrite operations in order to track progress	2017-07-18 13:30:29 -07:00
Paul Cruz	ae47eab2fd	changed test cases to use -s setting on the diffs	2017-07-18 12:58:50 -07:00
Stella Lau	fc41a87964	Experiment with using a lag when hashing	2017-07-17 18:13:09 -07:00
Paul Cruz	5af04c57b0	change parameters for compression level adapt	2017-07-17 17:59:50 -07:00
Paul Cruz	b3c9e02bb6	added signal to other threads whenever error occurs	2017-07-17 15:34:58 -07:00
Stella Lau	a00e406231	Remove version archive	2017-07-17 15:17:32 -07:00
Stella Lau	15a041adbf	Add function to get valid entries only from table	2017-07-17 15:16:58 -07:00
Paul Cruz	6be22f1f84	swap buffers instead of copying memory over	2017-07-17 14:39:10 -07:00
Paul Cruz	708238e07e	open file outside of adaptCCtx, pass to the output thread	2017-07-17 14:01:13 -07:00
Stella Lau	4bb42b02c1	Add basic chaining table	2017-07-17 11:53:54 -07:00
Paul Cruz	044e40db5a	removed freeCCtx() calls from createCCtx() so that it is not called twice during errors	2017-07-17 11:19:23 -07:00
Paul Cruz	50ce4eaeb6	added error detection for pthread initialization, added compression completion measurement, fixed const values	2017-07-17 10:12:44 -07:00
Stella Lau	ca300ce6e0	Decouple hash table from compression function	2017-07-14 17:17:00 -07:00
Paul Cruz	1ab3f06f00	updated tests to use different seeds when executing different tests	2017-07-14 16:29:29 -07:00
Stella Lau	6e443b4960	Move hash table access for own functions	2017-07-14 14:27:55 -07:00
Stella Lau	2d8e6c6608	Add more statistics	2017-07-14 12:31:01 -07:00
Stella Lau	55f960e8db	Add percentages to offset histogram	2017-07-14 11:00:20 -07:00
Stella Lau	4db7f12ef3	Add offset histogram	2017-07-14 10:52:03 -07:00
Paul Cruz	0c8b9436b7	removed goto statements for the most part	2017-07-13 16:38:20 -07:00
Stella Lau	175a6c6029	[ldm] Minor refactoring	2017-07-13 16:16:31 -07:00
Stella Lau	361c06df75	Add min/max offset to stats	2017-07-13 15:29:41 -07:00
Paul Cruz	65a4ce2635	added tests for forced compression level	2017-07-13 14:57:24 -07:00
Paul Cruz	0d9665cef5	added additional tests for performance, allowed force compression level for testing purposes	2017-07-13 14:46:54 -07:00
Stella Lau	2b3c7e4199	[ldm] Make some functions shared	2017-07-13 14:39:35 -07:00
Paul Cruz	9165e97fc6	added some tests for correctness, time, and compression ratio	2017-07-13 13:50:23 -07:00
Stella Lau	9306feb8fa	[ldm] Switch to using lib/common/mem.h and move typedefs to ldm.h ... Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Blame Revision:	2017-07-13 13:44:48 -07:00
Stella Lau	50421d9474	[ldm] Remove old main files	2017-07-13 11:45:00 -07:00
Stella Lau	68c4560701	[ldm] Add TODO and comment for segfaulting in compress function	2017-07-13 10:38:19 -07:00
Paul Cruz	766663f1f1	added altering dictionary size depending on compression level	2017-07-13 10:15:27 -07:00
Stella Lau	92bed4a7e0	[ldm] Add CHAR_OFFSET in hash function and extend header size	2017-07-12 18:47:26 -07:00
Paul Cruz	7c886db0a8	changed to stderr	2017-07-12 17:28:53 -07:00
Paul Cruz	b5b18cf664	changed to malloc, added comment about adaptive compression level, and changed ternary operators	2017-07-12 17:10:58 -07:00
Paul Cruz	954d999abf	fixed up freeCCtx() removed BYTE since it wasn't being used	2017-07-12 16:50:43 -07:00
Paul Cruz	3c16edd26a	added copyright header, removed clean from makefile	2017-07-12 16:40:24 -07:00
Stella Lau	8de82b6eb0	[ldm] Clean up versions	2017-07-12 16:31:31 -07:00
Paul Cruz	74d3a6f5ae	passes tests with adaptive compression level	2017-07-12 16:18:41 -07:00
Paul Cruz	5353d350ae	working with fixed compression level and fixed dictionary size	2017-07-12 16:02:20 -07:00
Stella Lau	8ff8cdb15b	[ldm] Clean up code	2017-07-12 15:12:07 -07:00
Paul Cruz	356ddb649f	working with flush job->src.size and fixed cLevel	2017-07-12 12:21:21 -07:00
Stella Lau	3a48ffd4fd	Fix sumToHash to use hash space more efficiently	2017-07-12 10:53:19 -07:00
Stella Lau	e0d4162464	Minor fix for non-rolling hash	2017-07-12 09:50:24 -07:00
Stella Lau	50502519fb	Switch to using rolling hash only	2017-07-12 09:47:00 -07:00
Stella Lau	583dda17a8	Update rolling hash	2017-07-11 18:13:26 -07:00
Paul Cruz	0a401852c4	added debug statement	2017-07-11 16:50:50 -07:00
Paul Cruz	72a183efad	changed dictionary size, added debugging statements	2017-07-11 15:49:52 -07:00
Paul Cruz	7c54e09347	updated DEBUG statements	2017-07-11 15:15:41 -07:00
Paul Cruz	a3c077b8c6	added error message, updated copying dictionary into the input buffer	2017-07-11 15:00:52 -07:00
Paul Cruz	34afb9b23e	changed to using ZSTD_compressBegin_usingDict() and fixed strange issue with ZSTD_compressContinue()	2017-07-11 11:50:00 -07:00
Paul Cruz	7ec5928626	fixed an error where -c argument wasn't working for single files	2017-07-11 10:23:25 -07:00
Stella Lau	f6c5d07fe2	Save v3	2017-07-11 09:23:44 -07:00
Stella Lau	6c3673f4c3	Add rolling hash	2017-07-10 22:27:43 -07:00
Paul Cruz	f918545491	made some progress on improving compression ratio, but problems exist with speed limits, and for some reason higher compression levels are really slow	2017-07-10 18:16:42 -07:00
Paul Cruz	01fc7c4244	changed how the detection of the last job works	2017-07-10 16:27:58 -07:00
Paul Cruz	c36552ef8a	dst buffer should use ZSTD_compressBound to determine how much space it needs	2017-07-10 16:10:19 -07:00
Paul Cruz	7aa36df6df	fixed memory leak that was happening when creating jobs	2017-07-10 16:03:09 -07:00
Stella Lau	ef2b728316	Clean up and refactor compress function	2017-07-10 15:48:47 -07:00
Paul Cruz	e410d63d45	made input buffer an internal part of the compression context	2017-07-10 15:37:14 -07:00
Stella Lau	e4155b11d7	Add warning flags to makefile and clean up code to remove warnings	2017-07-10 13:08:19 -07:00
Stella Lau	10a71d9f1c	Add compression context	2017-07-10 12:38:27 -07:00
Paul Cruz	cc7f8e4d71	small changes	2017-07-10 11:10:11 -07:00
Paul Cruz	7e09b508ff	changed name	2017-07-10 11:05:37 -07:00
Paul Cruz	ed72ea5438	removed single from Makefile	2017-07-10 10:58:03 -07:00
Paul Cruz	ced3ec5714	removed scripts	2017-07-10 10:53:02 -07:00
Paul Cruz	82f0d64bee	removed single.c	2017-07-10 10:51:50 -07:00
Paul Cruz	62ebbabd32	updated error checking in each thread	2017-07-10 09:36:22 -07:00
Stella Lau	ae9cf235d6	Add LDM_DCtx	2017-07-10 07:38:09 -07:00
Stella Lau	5432214ee3	Minor refactoring	2017-07-10 06:50:49 -07:00
Stella Lau	b94b468e84	Merge branch 'ldm' of https://github.com/stellamplau/zstd into ldm	2017-07-10 06:32:46 -07:00
Stella Lau	474e06ac5b	Minor refactoring	2017-07-10 06:32:29 -07:00
Stella Lau	eb280cd568	Add folder for old versions	2017-07-10 06:32:05 -07:00
Stella Lau	719ccdc5a5	Update mainfile	2017-07-09 22:45:54 -07:00
Stella Lau	acdeb9f302	Add compression statistics	2017-07-07 17:09:28 -07:00
Paul Cruz	c3ae23d459	added ability to compress without specifying out filename	2017-07-07 17:07:05 -07:00
Paul Cruz	7163ffafde	playing around with adapt param	2017-07-07 15:56:00 -07:00
Paul Cruz	1c9d6b2c6b	rewrote time elapsed with UTIL	2017-07-07 15:42:20 -07:00
Paul Cruz	c0c236a28b	changed to using compressCCtx	2017-07-07 15:13:40 -07:00
Stella Lau	4076be09ec	[ldm] Update to hash every position	2017-07-07 14:52:40 -07:00
Stella Lau	7945f9ee47	Fix offset overflow bug	2017-07-07 14:14:01 -07:00
Paul Cruz	11fc0f4119	changed completed -> compressed	2017-07-07 13:55:38 -07:00
Paul Cruz	09d7c6a994	changed completed variables to compressed for clarity	2017-07-07 13:18:55 -07:00
Stella Lau	f791fc27e3	Add header with compress and decompress size	2017-07-07 12:44:29 -07:00
Paul Cruz	8c0eb62920	removed unnecessary comments, uncommented DEBUGLOG for later use	2017-07-07 11:47:16 -07:00
Paul Cruz	70a4153bd3	added ability to force output to stdout, wrote an additional test for this functionality	2017-07-07 11:32:14 -07:00
Paul Cruz	532f439961	cleaned up code for arguments a bit	2017-07-07 10:58:43 -07:00
Paul Cruz	f7e6b358d0	added tests that check to ensure stdout is working	2017-07-07 10:29:06 -07:00
Paul Cruz	4679132f59	updated avg compression rate, also hiding progress bar behind a flag now	2017-07-07 10:25:38 -07:00
Paul Cruz	00bc5df4e0	added compression rate to status bar	2017-07-07 09:35:39 -07:00
Paul Cruz	f351848b76	added data amount	2017-07-06 20:40:00 -07:00