The Curse of Knowledge Humane Library Design with

The Curse of Knowledge Humane Library Design with D

The Curse of Knowledge • The curse of knowledge is a cognitive bias that occurs when an individual, communicating with other individuals, unknowingly assumes that the others have the background to understand. • Source: https: //en. wikipedia. org/wiki/Curse_of_knowledge

The Curse of Knowledge • Cognitive bias or blindness • The more you know the worse it becomes • It is exacerbated by intelligence • Simply knowing about it is not enough to counteract it

How We Design Libraries - The Laser Scalpel • I need to do _______ ! • It does everything I need it to do • Hey, this might be useful to someone else … • Git. Hub! • Result: The API surface is simple but only useful for the original use case

How We Design Libraries – The Hotpatch • Hey! This would be useful! • … • Result: Library does a (typically large) number of unrelated tasks.

How We Design Libraries – The Kitchen Sink • We don’t have ______ ! • Create an exhaustive wish list • Build everything on wish list • Result: Sprawling API with buggy implementations that cover every possible use case

The Curse Is A Lie Of Omission

Open. SSL – HMAC Implementation EVP_MD_CTX *mdctx = EVP_MD_CTX_create(); if (mdctx == null) throw new Cryptographic. Exception("Unable to create Open. SSL context. "); scope(exit) if(mdctx !is null) EVP_MD_CTX_destroy(mdctx); const(EVP_MD)* md = EVP_sha 384(); if (EVP_Digest. Init_ex(mdctx, md, null) != 1) throw new Cryptographic. Exception("Unable to create SHA-384 hash context. "); auto pkey = EVP_PKEY_new_mac_key(EVP_PKEY_HMAC, null, key. ptr, cast(int)key. length); scope(exit) if(pkey !is null) EVP_PKEY_free(pkey); if (EVP_Digest. Sign. Init(mdctx, null, md, null, pkey) != 1) throw new Cryptographic. Exception("Unable to create SHA-384 HMAC key context. "); if (EVP_Digest. Sign. Update(mdctx, data. ptr, data. length) != 1) throw new Cryptographic. Exception("Error while updating digest. "); ulong digestlen; ubyte[] digest = new ubyte[48]; if (EVP_Digest. Sign. Final(mdctx, digest. ptr, &digestlen) < 0) throw new Cryptographic. Exception("Error while retrieving the digest. ");

Botan – HMAC Implementation auto sha = new HMAC(new SHA 384()); scope(exit) sha. clear(); sha. set. Key(key. ptr, key. length); sha. update(data); auto digestvec = sha. finished(); ubyte[] digest = new ubyte[digestvec. length]; for(int i = 0; i<digestvec. length; i++) digest[i] = digestvec[i];

Better HMAC Implementation ubyte[] hash = hmac(key, data);

Secure. D API ubyte[] hash(ubyte[] data) public class Elliptic. Curve ubyte[] hash(string path) { ubyte[] hmac(ubyte[] key, ubyte[] data) this() ubyte[] hmac(ubyte[] key, string path) this(string private. Key, string password) ubyte[] pbkdf 2(ubyte[] key, string password, uint iterations = 25000, uint output. Len = 48) this(string public. Key) ubyte[] random(uint bytes) ~this() bool constant. Time. Equality(ubyte[] a, ubyte[] b) ubyte[] derive(string peer. Key) ubyte[] encrypt (ubyte[] key, ubyte[] data, ubyte[] additional. Data) ubyte[] sign(ubyte[] data, bool use. Sha 256 = false) bool validate (ubyte[] key, ubyte[] data, ubyte[] additional. Data) bool verify(ubyte[] data, ubyte[] signature, bool use. Sha 256 = false) ubyte[] decrypt (ubyte[] key, ubyte[] data, ubyte[] additional. Data) string get. Public. Key() string get. Private. Key(string password, int iterations = 25000, bool use 3 Des = false) }

Theoretical TLS API tls. Context handshake(ubyte[] data) ubyte[] encrypt(tls. Context ctx, ubyte[] data) ubyte[] decrypt(tls. Context ctx, ubyte[] data) void disconnect(tls. Context ctx, ubyte[] data) • Can work with any networking library • Simple API that clearly states what it is and how to use it

std. conv to(T)(T value) string num. Str = to!string(12345678);

Language == Library?

The D Language in Numbers • 112 Keywords • 63 Operators • 5 Global Symbols

assert • Check a conditional predicate • Write a message to stdout • Call terminate

foreach • Accept range as parameter • Pull item from range • Execute a delegate (the code inside the brackets) • Pull next item from range

pure • Evaluate code inside the brackets • Throw error if code modifies global state • List all possible escaping calls as errors

uint • Store value • Retrieve value • If value is greater than 4294967295, throw an overflow exception • If value is less than 0, throw an out-of-range exception

Reverse the Curse

Naming • Brevity for the sake of brevity is Evil • Use whole words where possible • writeln • How do non-native English speakers know? • Learn by rote • Slow uptake • writeln • write. Line

Naming • Abbreviations are acceptable when: • The abbreviation is well known (e. g. HTTP) • The expanded form is unwieldly (e. g. Hashed Message Authentication Code) • Avoid verbosity • zip. And. Upload. File. To. Aws. S 3 • write. Line. To. Console

Scope • Do one thing only • Do it well • Do it succinctly • Exception: Standard Libraries

Scope • Secure. D is 31 methods covering the following scenarios: • Symmetric Cryptography (AES-256 -CTR-HMAC) • Encryption • Authentication/Validation • Decryption • Asymmetric Cryptography (ECC-P 384 + RSA-AES-256 -CTR) • Load + Generate Public/Private Keys • Key Derivation • Sign + Verify • Hashing (SHA 2 -384) • Regular and Keyed (HMAC) Hashing • File Hashing • Password Based Key Derivation (PBKDF 2) • Secure. D implements Open. SSL and Botan via DUB configuration • Method signatures do not change per implementation

Modularity • Almost any library of value will have more than one source file. • In D each file is a module. • Secure. D is 9 modules and 1 package module. • Each Module can be imported individually • import secured. aes; • A Package Module can be used to import a list of modules • Packages do not need to include all modules • Secure. D does not include rsa. d in the package

Modularity - Packages module secured; public import secured. aes; public import secured. ecc; public import secured. hash; public import secured. hmac; public import secured. kdf; public import secured. random; public import secured. util; import secured;

Overloading • Avoid overloading intent • Overloads are variants of a single inrent • Overloading intent is guaranteed to cause confusion at the call-site • Overloading language constructs is EVIL • Especially when the intents are tenuously related

Overloading • Choose parameters that will be obvious when reading the call-site • auto digest = hmac(key, password. Bytes); • auto digest = hmac(key, “/usr/libssl. dylib”); • Templates are enormously powerful • But with great power comes great responsibility • Templates are overloading on steroids

Configuration • Secure. D uses version to compile code for either Open. SSL or Botan • version is a special case of static if • • Analogous to #define but MUCH cleaner Compiler is passed a -version= string Code inside a version block not matching the -version= string is not compiled Enables supporting multiple dependent libraries in the same library

Deprecation • Provides a method of removing broken or badly designed code • deprecated keyword in D • • Provide an alternative Can have an optional message associated with it. Provide a clear removal date in message and release notes If you do not have a clear removal date, do not deprecate the feature • Usage: • deprecated(“This method has been deprecated and will be removed in the 12/17 release. Please use <othermethod> instead. ”)

Sanity Checking • in { } and out { } • in { } • Specialized code can be run to ensure the sanity of incoming parameters • Secure. D uses in { } to enforce parameter lengths and values • assert(key. length == 32, "Encryption key must be 32 bytes in length. "); • out { } • Used to determine if the return value is invalid

Unit Testing • unittest { } • Built into the language • Tests can be run as part of every build • Secure. D has Unit Tests for each method in the library • For open source libraries, the engineer can run the tests to verify the library returns the expected results. • Secure. D uses unittests extensively to verify correctness

Existing Libraries • Use the built-in deprecation mechanism • Your language has one right? • Wrap old methods in new names • Provide new, simpler methods • Once the deprecation period has passed, remove the old method • A defined deprecation strategy smoothens out the change • But what about <insert important user here>? • Chances are they’ve been begging for this • The complexity in your API is maintenance overhead in theirs • Breaking changes scare off users! • Existing users will complain loudly, then use the new API • New users will find your library more approachable

But Where are the Templates? • Templates are overloading gone wild. • Now types are overloaded as well as method signatures. • Notoriously difficult to debug in C++ • Easier to debug in D • Use only as appropriate • • Secure. D has no need for any of the capabilities of templates Use only as often as strictly necessary Limit to one template parameter as much as possible to(T) in std. conv is my favorite example of a well designed template • to!string(12345678);

<Insert Your Library Here> • Avoid overloaded methods • Prefer human friendly names • “Stay on target!” • A library that implements crypto should not implement string processing • A library that processes HTTP requests should not implement crypto • Choose composition over implementation • Don’t reinvent the wheel • Take dependencies on existing packages were available • DUB is your friend

Phobos • D Standard Library • Wish List • Split each namespace (std. *) into a seperate DUB package • Module interdependencies become extremely clear • Select-a-size model for Phobos • • Entire Library Package Rings (Base, No. GC, Complete, Experimental) Package Groups (System, Console, Files, Strings, Ranges, Containers, Data) Single Module (std. base 64, etc. )

It’s Ok To… • Remove Options • Provide thoughtful defaults • Allow customizability at key points • Most engineers don’t want to think about the choices anyways • Take Dependencies • Reduces complexity of your code • Any decent build system can handle them, DUB makes them trivial • Allows you to focus on the problem you are trying to solve

Desired Outcomes • Improved Accessibility • Newcomers will learn the library faster • Engineers can quickly select the desired effect • Maintainers will find it easier to maintain code written by others • Reduced Complexity • Decreases time to working knowledge • Decreases maintenance cost • Decreases time-to-ship

Desired Outcomes • Principle of Single Responsibility • Do one thing and do it will • Reduces complexity of both the code and the API surface • Standard Libraries are exempted • Modularity • Allow engineer to pay for only what they need • Provide Package Modules • Include both top-level and sub-level package modules

Review • The Code • Avoid advanced language features • Design for modularity • The API • Simple and Intuitive • Open. SSL is the perfect example of what not to do • Avoid advanced language features • Hide complexity • Open. SSL again • A complex problem does not typically require a complex solution

Contact • Email + DLang Forums: flyboynw@gmail. com • IRC: Light. Bender • Github: https: //github. com/lightbender • Secure. D: https: //github. com/lightbender/secured • DLang Package Repository: https: //code. dlang. org/ • DUB: https: //code. dlang. org/download