Into the Tar Pit A Tar MK Deep

APACHE SLING & FRIENDS TECH MEETUP BERLIN, 26 -28 SEPTEMBER 2016 Into the Tar

Tar. MK Application Oak JCR AEM Sling OSGi Oak Core Content Repository Servlet Engine

Tar. MK § Embedded Database § § Hierarchical Fast / Small Limited scalability MVCC

Agenda § § MVCC Persistence Revisions, Recovery, Rollback Garbage Collection Upcoming Improvements © 2017

MVCC Persistence © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Updating Trees A A’ A B D B C E F © 2017 Adobe

Revisions HEAD R 1 HEAD R 2 © 2017 Adobe Systems Incorporated. All Rights

Persisting Revisions A B D C E F © 2017 Adobe Systems Incorporated. All

Records and Segments 7 d 78 a 945 -4553 -409 b-adad 7050256 c 05

Tar Files data 00000 a. tar data 00001 a. tar data 00002 b. tar

Revisions, Recovery, Rollback © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Recovery $ ls segmentstore 256 M Aug 16 17: 09 data 00000 a. tar

Recovery 17: 12: 21. 025 WARN Could not find a valid tar index in

Revisions $ cat journal. log fd 155 d 2 d-516 c-4274 -aa 83 -0851

Rollback $ java -jar oak-run-*. jar check usage: check <options> Option Description ---------bin [Long]

Rollback $ java -jar oak-run-*. jar check --deep --path /segmentstore 21: 52: 07. 149

Rollback 21: 52: 07. 228 INFO Checking revision bb 8 b 37 a 3

Garbage Collection © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Offline Revisions GC R 1 R 2 © 2017 Adobe Systems Incorporated. All Rights

Offline Revision GC $ java –jar oak-run-*. jar compact segmentstore Compacting segmenstore before Tue

Offline Revision GC -> writing new journal. log: 3 b 632859 -fafd-4113 -a 53

Online Revisions GC § Same as Offline § BUT Expensive § Huge and dense

Online Revisions GC: Roots R 1 R 2 R 3 Session 1 Session 2

Upcoming Improvements © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Online Revisions GC § Retention by Generation § No gc roots from compacted head

and beyond. . . § Partial and background gc § Scalable § Resumable §

Conclusion § Many improvements for gc § § Changed storage format Migration required §

Questions © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Slides: 29

Download presentation

Into the Tar Pit: A Tar. MK Deep Dive Michael Dürig, Adobe

APACHE SLING & FRIENDS TECH MEETUP BERLIN, 26 -28 SEPTEMBER 2016 Into the Tar Pit: A Tar. MK Deep Dive Michael Dürig, Adobe Research

Records and Segments 7 d 78 a 945 -4553 -409 b-adad 7050256 c 05 fe D E B cbde 7 a 01 -b 30 c-4716 -a 9 a 729 d 7 ce 1 f 5838 F C A © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. … 57 c 76710 -a 690 -4028 -a 498 -26477293 b 5 … … …

Tar Files data 00000 a. tar data 00001 a. tar data 00002 b. tar 57 c 76710 -a 690 -4028 -a 498 -26477293 b 5 6 e 162 b 11 -3782 -47 ca-a 78 d 4 da 12149 df 8 d 7911 f 3 af-a 286 -4 c 4 f-a 944 -8 ed 235 c 723 e 1 2236063 c-77 a 0 -47 b 0 -a 16 a 30 f 4899990 a 4 9 e 884 e 53 -b 1 b 2 -4906 -a 0 a 73 a 51 c 77579 fd e 098 df 2 a-3958 -4 d 4 b-a 651 -6 b 8498 c 22 f 66 d 9 f 5 b 47 d-0 fea-4 a 77 -aa 8 e-3 fdc 0302 ef 24 090 e 4312 -a 115 -44 e 8 -ab 470 a 89 f 380 ab 64 67 d 9 fc 69 -d 2 d 6 -4543 -a 189 d 24 d 4 c 00 db 67 20 bffcd 4 -9 d 48 -46 ca-a 263 -fa 49 ba 1 d 8 cb 3 f 2178987 -09 d 2 -48 de-abc 77718 dc 8 b 8 c 74 ec 4 e 2563 -7 d 2 e-4 c 54 -a 52 f-9582 c 3 a 6 fb 54 1 d 32 af 7 d-3 c 1 c-4892 -a 44 ae 4 a 5 d 4 e 33 7 e 68 db 78 -3 aca-4 a 34 -a 72 f-c 174 e 8 f 8 c 93 d f 32 f 6 bf 8 -c 7 cc-4 e 20 -aa 94 -d 2 e 783 bf 76 d 5 © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Recovery $ ls segmentstore 256 M Aug 16 17: 09 data 00000 a. tar 256 M Aug 16 17: 09 data 00001 a. tar 256 M Aug 16 17: 09 data 00002 a. tar $256 M tar Aug –tvf 16 data 0000 a. tar 17: 09 data 00003 a. tar 69644 Aug 16 17: 09 data 00004 a. tar 0686 c 08 d-e 3 f 6 -474 e-bb 32 -874 efca 706 e 7. 58 dbce 7 b 256 M Aug 16 17: 09 262144 Aug 16 1617: 09 data 00005 a. tar a 3 a 57501 -f 30 f-4986 -b 820 -e 166 c 50 adaad. 8 d 58 d 425 256 M Aug 262144 Aug 16 1617: 09 data 00006 a. tar 8 bf 5 c 193 -6458 -4 e 29 -b 4 f 5 -d 0 dbba 5 ca 584. 0 a 0 ceeac 256 M Aug 195080 Aug 16 1617: 12 17: 09 data 00007 a. tar 2 f 6 cbfdf-8 d 78 -41 d 9 -b 507 -89 f 47 a 143 cab. 47624 a 71 231 M Aug 262144 Aug 16 1617: 09 data 00007 a. tar. bak 7 b 8 fd 991 -e 894 -49 fb-b 0 e 1 -c 193 e 5403755. da 89 a 3 db 231 M Aug 262144 Aug 16 1617: 12 17: 09 data 00008 a. tar 10 d 3 ea 6 c-e 62 f-45 d 1 -bd 61 -fab 94 db 9 cddd. da 137 b 63 256 M Aug 25600 Aug 16 1617: 12 17: 09 data 00009 a. tar data 00000 a. tar. gph 182 M 31232 Aug 16 1617: 09 journal. log data 00000 a. tar. idx 147 B © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Recovery 17: 12: 21. 025 WARN Could not find a valid tar index in [/segmentstore/data 00007 a. tar], recovering. . . 17: 12: 21. 025 INFO Recovering segments from tar file /segmentstore/data 00007 a. tar 17: 12: 21. 739 INFO Backing up /segmentstore/data 00007 a. tar to data 00007 a. tar. bak 17: 12: 21. 739 INFO Regenerating tar file /segmentstore/data 00007 a. tar © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Revisions $ cat journal. log fd 155 d 2 d-516 c-4274 -aa 83 -0851 bbc 2 eb 47: 102112 root bb 8 b 37 a 3 -8129 -45 b 7 -a 043 -484 a 299523 da: 182460 root 639 b 7832 -7 fcc-4 f 42 -abe 7 -48 c 4 f 5505850: 162320 root R 1 HEAD R 2 © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. HEAD

Rollback $ java -jar oak-run-*. jar check usage: check <options> Option Description ---------bin [Long] read the n first bytes from binary properties. -1 for all bytes. (default: 0) --deep [Long] enable deep consistency checking. An optional long specifies the number of seconds between progress notifications (default: 9223372036854775807) --journal file (default: journal. log) --path to the segment store (required) © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Rollback $ java -jar oak-run-*. jar check --deep --path /segmentstore 21: 52: 07. 149 INFO Searching for last good revision in journal. log 21: 52: 07. 219 INFO Checking revision 639 b 7832 -7 fcc-4 f 42 -abe 748 c 4 f 5505850: 162320 21: 52: 07. 227 ERROR Segment not found: 639 b 7832 -7 fcc-4 f 42 -abe 748 c 4 f 5505850. Creation date delta is 6 ms. 21: 52: 07. 227 INFO Error while traversing 639 b 7832 -7 fcc-4 f 42 -abe 748 c 4 f 5505850: 162320 21: 52: 07. 228 INFO Broken revision 639 b 7832 -7 fcc-4 f 42 -abe 748 c 4 f 5505850: 162320 © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Rollback 21: 52: 07. 228 INFO Checking revision bb 8 b 37 a 3 -8129 -45 b 7 -a 043484 a 299523 da: 182460 21: 52: 07. 228 INFO Checking / 21: 52: 07. 266 INFO Traversed 88 nodes and 103 properties 21: 52: 07. 266 INFO Found latest good revision bb 8 b 37 a 3 -8129 -45 b 7 -a 043484 a 299523 da: 182460 21: 52: 07. 266 INFO Searched through 2 revisions © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Offline Revision GC $ java –jar oak-run-*. jar compact segmentstore Compacting segmenstore before Tue Aug 16 17: 09: 05 CEST 2016, data 00000 a. tar Tue Aug 16 17: 09: 08 CEST 2016, data 00001 a. tar. . . size 2. 6 GB (2582279827 bytes) -> compacting -> cleaning up -> removed old file data 00000 a. tar -> removed old file data 00001 a. tar. . . © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Offline Revision GC -> writing new journal. log: 3 b 632859 -fafd-4113 -a 53 a-335451933862: 231132 root after Tue Aug 23 11: 45: 08 CEST 2016, data 00000 b. tar Tue Aug 23 11: 45: 08 CEST 2016, data 00001 b. tar. . . size 546. 4 MB (546363953 bytes) removed files [data 00000 a. tar, . . . ] added files [data 00000 b. tar, . . . ] Compaction succeeded in 4. 240 s (4 s). © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Online Revisions GC § Same as Offline § BUT Expensive § Huge and dense reference graphs Resource Contender § Concurrent writes § CPU § IO § Heap § Locks § Compacted§head Cache coherence / locality Additional GC roots © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Online Revisions GC § Retention by Generation § No gc roots from compacted head § Ignore gc roots from heap § Based on retention time instead § Cheaper than by reference © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Conclusion § Many improvements for gc § § Changed storage format Migration required § § § More scalable gc Bigger repositories More write throughput § Ground work for future improvements © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.