DESIGN OF KOTLINX SERIALIZATION LEONID STARTSEV sandwwraith Copenhagen

DESIGN OF KOTLINX. SERIALIZATION LEONID STARTSEV @sandwwraith Copenhagen Denmark

At a first glance, it looks like a regular JSON library 2

At a first glance, it looks like a regular JSON library val json = Json(Json. Configuration. Stable) 3

At a first glance, it looks like a regular JSON library val json = Json(Json. Configuration. Stable) json. parse(. . . ) json. stringify(. . . ) 4

Why do we need yet another json parser? 5

Why do we need yet another json parser? These all are Java libraries 6

Multiplatform @Serializable data class Session( val id: String, val is. Service. Session: Boolean, val is. Plenum. Session: Boolean, val question. Answers: List<Question. Answer>, val speakers: List<String>, @Serial. Name("description") val description. Text: String? , val starts. At: String? , val title: String, val ends. At: String? , val category. Items: List<Int>, val room. Id: Int? ) 7

Kotlin-oriented data class Data( val data: String, val optional. Data: String = "empty" ) const val input. String = """{"data": "Foo"}""" 8

Kotlin-oriented data class Data( val data: String, val optional. Data: String = "empty" ) val data = Gson(). from. Json(input. String, Data: : class. java) // Data(data=Foo, optional. Data=null) Null for non-nullable type, definitely an error 9

Kotlin-oriented @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) val data = Json. parse(Data. serializer(), input. String) // Data(data=Foo, optional. Data=empty) OK! 10

Explicit @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) 11

Explicit @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) val serializer: KSerializer<Data> = Data. Companion. serializer() 12

Explicit @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) val serializer: KSerializer<Data> = Data. Companion. serializer() val data = Json. parse(Data. serializer(), input. String) 13

Implicit requires opt-in val data = Json. parse<List<Data>>(input. String) 14

Implicit requires opt-in val data = Json. parse<List<Data>>(input. String) This declaration is experimental and its usage must be marked with @Use. Experimental(Implicit. Reflection. Serializer: : class) 15

Implicit requires opt-in val data = Json. parse<List<Data>>(input. String) This declaration is experimental and its usage must be marked with @Use. Experimental(Implicit. Reflection. Serializer: : class) ● Doesn’t work on Native ● Doesn’t work for collections 16 https: //github. com/Kotlin/kotlinx. serialization/blob/master/runtime/common. Main/src/kotlinx/serialization/Serial. Implicits. kt#L 8

Concise val data: List<Data> = Gson(). from. Json( ) 17

Concise val data: List<Data> = Gson(). from. Json( input. Generic. String, (object: Type. Token<List<Data>>() {}). type ) A lot of ceremony 18

Concise val data: List<Data> = Json. parse( Data. serializer(). list, ) input. Generic. String Short and uniform 19 https: //github. com/Kotlin/kotlinx. serialization/blob/master/docs/runtime_usage. md#obtaining-serializers

Explicit & concise & compiletime safe @Serializable data class Complex. Data( val description: String, val data: Complex. Class ) 20

Explicit & concise & compiletime safe @Serializable data class Complex. Data( val description: String, val data: Complex. Class ) Serializer for type ‘Complex. Class’ has not been found IDE diagnostic! (from Kotlin 1. 3. 50) 21

Formats are pluggable val string. Data = Json. stringify(Data. serializer(), data) val bytes. Data 22 = Proto. Buf. dump (Data. serializer(), data)

Formats are pluggable val string. Data = Json. stringify(Data. serializer(), data) val bytes. Data = Proto. Buf. dump (Data. serializer(), data) Different formats, same API 23

Difference between serializer and format 24

Difference between serializer and format Stream of primitives Kotlin class Serialization by serializer 25 Output stream Encoding by format

Difference between serializer and format ● Optionals handling ● Class metadata Stream of primitives Kotlin class Serialization by serializer 26 Output stream Encoding by format

Difference between serializer and format ● Optionals handling ● Class metadata Stream of primitives Kotlin class Serialization by serializer 27 ● Text or bytes ● Versioning ● Schema or schemaless Output stream Encoding by format

Formats have their specific configuration val json = Json(Json. Configuration( encode. Defaults = false, unquoted = true, pretty. Print = true )) 28

Formats have their specific configuration val json = Json(Json. Configuration( encode. Defaults = false, unquoted = true, pretty. Print = true )) val protobuf = Proto. Buf() 29

Part II Behind the scenes

Why @Serializable and apply plugin: ‘. . . ’? @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) 31

Why @Serializable and apply plugin: ‘. . . ’? @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) // @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) 32

Why @Serializable and apply plugin: ‘. . . ’? apply plugin: 'kotlin-multiplatform' //apply plugin: 'kotlinx-serialization' 33

Why @Serializable and apply plugin: ‘. . . ’? apply plugin: 'kotlin-multiplatform' apply plugin: 'kotlinx-serialization' Kotlin serialization consists of acompiler plugin, which automatically produces visitor code for classes, and runtime library, which uses generated code to serialize objects without reflection. 34

What is a compiler plugin, anyway? 35

What is a compiler plugin, anyway? MAGIC 36

What is a compiler plugin, anyway? Frontend (parsing, resolution) JVM Backend 37 JS Backend Very simple scheme of Kotlin compiler Native Backend

Serialization plugin Adds. serializer() function Frontend (parsing, resolution) JVM Backend 38 JS Backend Not so simple scheme of Kotlin compiler Native Backend

Serialization plugin Adds. serializer() function Frontend (parsing, resolution) Additional code generation JVM Backend 39 JS Backend Not so simple scheme of Kotlin compiler Native Backend

So we are writing a serializer, right? 40

Difference between serializer and format Stream of primitives Kotlin class Serialization by serializer 41 Output stream Encoding by format

Which methods does Encoder have? interface Encoder { fun encode. Not. Null. Mark() fun encode. Null() fun fun fun encode. Unit() encode. Boolean(value: Boolean) encode. Byte(value: Byte) encode. Short(value: Short) encode. Int(value: Int) encode. Long(value: Long) encode. Float(value: Float) encode. Double(value: Double) encode. Char(value: Char) fun encode. String(value: String) fun encode. Enum(enum. Description: Enum. Descriptor, ordinal: Int) } 42 fun <T : Any? > encode. Serializable. Value(serializer: Serialization. Strategy<T>, value: T)

Which methods does Encoder have? interface Encoder { fun encode. Not. Null. Mark() fun encode. Null() fun fun fun encode. Unit() encode. Boolean(value: Boolean) encode. Byte(value: Byte) encode. Short(value: Short) encode. Int(value: Int) encode. Long(value: Long) encode. Float(value: Float) encode. Double(value: Double) encode. Char(value: Char) One function for each ‘primitive’ data type fun encode. String(value: String) fun encode. Enum(enum. Description: Enum. Descriptor, ordinal: Int) } 43 fun <T : Any? > encode. Serializable. Value(serializer: Serialization. Strategy<T>, value: T)

Transform to single value @Serializable data class Color(val r: Int, val g: Int, val b: Int) 44

Transform to single value @Serializable data class Color(val r: Int, val g: Int, val b: Int) "#00 FFAA" 45

Transform to single value object Single. Color. Serializer : KSerializer<Color> { override fun serialize(encoder: Encoder, obj: Color) { } } 46

Transform to single value object Single. Color. Serializer : KSerializer<Color> { override fun serialize(encoder: Encoder, obj: Color) { val code: String = obj. r. to. Hex. Code() + obj. g. to. Hex. Code() + obj. b. to. Hex. Code() } } 47

Transform to single value object Single. Color. Serializer : KSerializer<Color> { override fun serialize(encoder: Encoder, obj: Color) { val code: String = obj. r. to. Hex. Code() + obj. g. to. Hex. Code() + obj. b. to. Hex. Code() encoder. encode. String("#$code") } } 48

Transform from multiple values """{"red": 255, "green": 128, "blue": 0}""" 49

Transform from multiple values """{"red": 255, "green": 128, "blue": 0}""" @Serializable data class Color(val argb: Int) 50

Transform from multiple values override fun deserialize(decoder: Decoder): Color { var result = 0 result += decoder. decode. Int() shl 16 result += decoder. decode. Int() shl 8 result += decoder. decode. Int() shl 0 return Color(result) } 51

Transform from multiple values override fun deserialize(decoder: Decoder): Color { var result = 0 result += decoder. decode. Int() shl 16 result += decoder. decode. Int() shl 8 result += decoder. decode. Int() shl 0 return Color(result) } We don’t know whether ‘red’ goes first or not 52

How to handle structured data interface Composite. Encoder { fun encode. Int. Element(desc: Serial. Descriptor, index: Int, value: Int). . . fun encode. String. Element(desc: Serial. Descriptor, index: Int, value: String) } Containing declaration 53 Position in structure https: //github. com/Kotlin/KEEP/blob/master/proposals/extensions/serialization. md#encoder-interfaces

Transform from multiple values object Multiple. Color. Serializer : KSerializer<Color> { override fun deserialize(decoder: Decoder): Color { var result = 0 } 54 } return Color(result)

Transform from multiple values object Multiple. Color. Serializer : KSerializer<Color> { override fun deserialize(decoder: Decoder): Color { var result = 0 val decoder: Composite. Decoder = decoder. begin. Structure(descriptor) decoder. end. Structure(descriptor) return Color(result) } } 55

Transform from multiple values override fun deserialize(decoder: Decoder): Color { var result = 0 val decoder = decoder. begin. Structure(descriptor) while (true) { when (val i = decoder. decode. Element. Index(descriptor)) { } } 56 } decoder. end. Structure(descriptor) return Color(result)

Transform from multiple values override fun deserialize(decoder: Decoder): Color { var result = 0 val decoder = decoder. begin. Structure(descriptor) } 57 loop@ while (true) { when (val i = decoder. decode. Element. Index(descriptor)) { 0 -> result += decoder. decode. Int. Element(descriptor, i) shl 16 1 -> result += decoder. decode. Int. Element(descriptor, i) shl 8 2 -> result += decoder. decode. Int. Element(descriptor, i) shl 0 READ_DONE -> break@loop } } decoder. end. Structure(descriptor) return Color(result)

Difference between serializer and format Stream of primitives Kotlin class Serialization by serializer Can be auto-generated! 58 Output stream Encoding by format

How KSerializer looks inside? @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) 59

How KSerializer looks inside? 60

How KSerializer looks inside? 61 https: //github. com/Kotlin/KEEP/blob/master/proposals/extensions/serialization. md#code-generation

How KSerializer looks inside? . deserialize call begin. Structure decode. Element. Index decode[Type]Element 62

How KSerializer looks inside? . deserialize call begin. Structure decode. Element. Index decode[Type]Element end. Structure 63 Constructor call

Serial. Descriptor 64

Serial. Descriptor @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) 65

Serial. Descriptor @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) name 66 "com. myapp. Data"

Serial. Descriptor @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) Contains useful meta information 67 name "com. myapp. Data" elements[0]. name "data" elements[1]. name "optional. Data" elements[1]. is. Optional true

Part III Diving into encoding

Format interfaces don’t tell you anything about… format override fun encode. Int(value: Int) = ? ? ? override fun encode. String(value: String) = ? ? ? 69

Serializable = Parcelable? class Parcelizer(val bundle: Bundle): Named. Value. Encoder() { } 70

Serializable = Parcelable? class Parcelizer(val bundle: Bundle): Named. Value. Encoder() { override fun encode. Tagged. Int(tag: String, value: Int) { } override fun encode. Tagged. String(tag: String, value: String) { } } 71

Serializable = Parcelable class Parcelizer(val bundle: Bundle): Named. Value. Encoder() { override fun encode. Tagged. Int(tag: String, value: Int) { bundle. put. Int(tag, value) } override fun encode. Tagged. String(tag: String, value: String) { bundle. put. String(tag, value) } } 72

Serializable = Multiplatform Parcelable expect class Bundle actual typealias Bundle = android. os. Bundle class Parcelizer(val bundle: Bundle): Named. Value. Encoder() { override fun encode. Tagged. Int(tag: String, value: Int) { bundle. put. Int(tag, value) } override fun encode. Tagged. String(tag: String, value: String) { bundle. put. String(tag, value) } } 73

You even don’t have to store values anywhere 74

You even don’t have to store values anywhere val digest = Message. Digest. get. Instance("MD 5") 75

You even don’t have to store values anywhere val digest = Message. Digest. get. Instance("MD 5") class Hasher(val digest: Message. Digest): Element. Value. Encoder() { } 76

You even don’t have to store values anywhere val digest = Message. Digest. get. Instance("MD 5") class Hasher(val digest: Message. Digest): Element. Value. Encoder() { override fun encode. Byte(value: Byte) { digest. update(value) } override fun encode. String(value: String) { digest. update(value. to. Byte. Array()) } } 77

ORM done wrong class Statement. Serializer(val stmt: Prepared. Statement): Element. Value. Encoder() { } 78

ORM done wrong class Statement. Serializer(val stmt: Prepared. Statement): Element. Value. Encoder() { override fun encode. String(value: String) { stmt. set. String(i++, value) } } 79

ORM done wrong class Statement. Serializer(val stmt: Prepared. Statement): Element. Value. Encoder() { override fun encode. String(value: String) { stmt. set. String(i++, value) } } class Result. Set. Deserializer(private val rs: Result. Set): Named. Value. Decoder() { } 80

ORM done wrong class Statement. Serializer(val stmt: Prepared. Statement): Element. Value. Encoder() { override fun encode. String(value: String) { stmt. set. String(i++, value) } } class Result. Set. Deserializer(private val rs: Result. Set): Named. Value. Decoder() { override fun decode. Tagged. String(tag: String): String { return rs. get. String(tag) } } 81

Serial. Descriptor @Serializable data class Data( val data: String, val optional. Data: String = "empty" ) Contains useful meta information 82 name "com. myapp. Data" elements[0]. name "data" elements[1]. name "optional. Data" elements[1]. is. Optional true

Serial. Descriptor @Serializable data class Generic. Data<T>( val description: String, val content: T ) 83

Serial. Descriptor @Serializable data class Generic. Data<T>( val description: String, val content: T ) Generic. Data. serializer()). descriptor. dump() 84

Serial. Descriptor Generic. Data content data 85 description optional. Data

Serial. Descriptor Generic. Data com. myapp. Generic. Data { description : kotlin. String content : com. myapp. Data { data : kotlin. String optional. Data : kotlin. String } } content data 86 description optional. Data

JSON Schema 87

JSON Schema fun Json. Schema(descriptor: Serial. Descriptor): Json. Object { } 88

JSON Schema fun Json. Schema(descriptor: Serial. Descriptor): Json. Object { descriptor. element. Descriptors(). for. Each. Indexed { index, child -> } } 89

JSON Schema fun Json. Schema(descriptor: Serial. Descriptor): Json. Object { } 90 descriptor. element. Descriptors(). for. Each. Indexed { index, child -> val element. Name = descriptor. get. Element. Name(index) properties[element. Name] = Json. Schema(child) }

JSON Schema fun Json. Schema(descriptor: Serial. Descriptor): Json. Object { descriptor. element. Descriptors(). for. Each. Indexed { index, child -> val element. Name = descriptor. get. Element. Name(index) properties[element. Name] = Json. Schema(child) } } 91 val object. Data = mutable. Map. Of( "description" to Json. Literal(descriptor. name), "properties" to Json. Object(properties) ) return Json. Object(object. Data) https: //github. com/Kotlin/kotlinx. serialization/blob/master/examples/example-visitors/src/main/kotlinx/serialization/schema/Json. Schema. kt

@Serializable data class Data(val data: String, val optional. Data: String = "empty") { 92 } "description": "com. myapp. Data", "properties": { "data": { "description": "kotlin. String" }, "optional. Data": { "description": "kotlin. String" } }

Roadmap

Roadmap ✓ Core ✓ Meta information in descriptors ✓ Multiplatform polymorphism ✓ IDE diagnostics ✓ Objects and sealed classes ✓ Done

Roadmap ✓ Core ✓ Done ✓ Meta information in descriptors ✓ Multiplatform polymorphism ✓ IDE diagnostics ✓ Objects and sealed classes ➢ Inline classes ➢ In progress

Roadmap ✓ Core ✓ Done ✓ Meta information in descriptors ✓ Multiplatform polymorphism ✓ IDE diagnostics ✓ Objects and sealed classes ➢ Inline classes ➢ In progress IO Streaming & Performance tuning Planned

Roadmap ✓ Core ✓ Done ✓ Meta information in descriptors ✓ Multiplatform polymorphism ✓ IDE diagnostics ✓ Objects and sealed classes ➢ Inline classes ➢ In progress IO Streaming & Performance tuning Planned Out of experimental Kotlin 1. 4 Community driven formats

THANK YOU AND REMEMBER TO VOTE https: //github. com/Kotlin/ kotlinx. serialization Leonid Startsev@sandwwraith #Kotlin. Conf