A Principled Approach to Graph QL Query Cost

A Principled Approach to Graph. QL Query Cost Analysis Alan Cha, Erik Wittern, Guillaume Baudart, James C. Davis, Louis Mandel, Jim A. Laredo Now a professor at:

Graph. QL is a query language for APIs and a runtime for fulfilling those queries. . . 2

Hypothetical REST API Task: get Alan’s friends’ names GET /users/alan_cha { first. Name: “Alan”, last. Name: “Cha”, friend. Ids: [“erik-wittern“, “gbaudart”, “James. D 123”], interests: [“graphql”, “programming”] } GET /users/erik-wittern { GET /users/gbaudart { first. Name: “Erik”, last. Name: “Wittern”, friend. Ids: [“. . . ”], interests: [“. . . ”] } { first. Name: “Guillaume”, last. Name: “Baudart”, friend. Ids: [“. . . ”], interests: [“. . . ”] } 3 GET /users/James. D 123 first. Name: “James”, last. Name: “Davis”, friend. Ids: [“. . . ”], interests: [“. . . ”] }

Hypothetical Graph. QL API Task: get Alan’s friends’ names { user: { friends: [ { first. Name: “Erik”, last. Name: “Wittern” }, { first. Name: “Guillaume”, last. Name: “Baudart” }, { first. Name: “James”, last. Name: “Davis” } ] } POST /graphql { user(user. Id: “alan_cha”) { friends { first. Name last. Name } } 4

Comparison Graph. QL… ● reduces round trips ● limits overfetching ● minimizes parsing REST API Graph. QL API GET /users/alan_cha { user: { friends: [ { first. Name: “. . . ”, last. Name: “. . . ” }, { first. Name: “. . . ”, last. Name: “. . . ” } ] } { first. Name: “Alan”, last. Name: “Cha”, friend. Ids: [“. . . ”, “. . . ”], interests: [“. . . ”, “. . . ”] POST /graphql { user(user. Id: “. . . ”) { friends { first. Name last. Name } } } GET /users/erikwittern { GET /users/gbaudart { first. Name: “. . . ”, last. Name: “. . . ”, friend. Ids: [“. . . ”], interests: [“. . . ”] } } { first. Name: “. . . ”, last. Name: “. . . ”, friend. Ids: [“. . . ”], interests: [“. . . ”] } 5 GET /users/James. D 123 first. Name: “. . . ”, last. Name: “. . . ”, friend. Ids: [“. . . ”], interests: [“. . . ”] } }

6

7

High-cost Graph. QL queries POST /graphql { user(user. Id: “. . . ”) { friends { first. Name last. Name } } POST /graphql { POST /graphql user(user. Id: “. . . ”) { friends { first. Name last. Name } } } +3 { user(user. Id: “. . . ”) { friends { first. Name last. Name } } } +3 } } 1 root user + 3 friends = 4 users 8 +9 1 root user + 3 friends + 3 x 3 friends of friends = 13 users +27 1 root user + 3 friends + 3 x 3 friends of friends = 40 users

Query analysis survey We studied the documentation for 30 public Graph. QL APIs listed on APIs. guru • 25 APIs (83%) do not describe any query analysis • 22 APIs (73%) do not perform rate limiting • 3 APIs (10%) perform rate limiting only by request frequency Many public Graph. QL APIs do not document any query analysis 9

Existing query analyses Dynamic • • Obtains list sizes during execution Additional runtime load Engineering costs for integration Accurate estimate 10 Static • • Assumes a pathological data graph Relies on configuration for list sizes No backend integration Estimate is an upper-bound Accurate Fast Too costly Generalizable Inflexible Primitive

Contributions 1. Two distinct definitions of query complexity 2. Analysis can handle common schema conventions 3. Analysis built on formal Graph. QL semantics 4. First evaluation of such an analysis on real-world APIs 11

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } 12 Query { Response { user(user. Id: 1) { name } } "user": { "name": "Alan Cha" } }

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } { Response { user(user. Id: 1) { name } } Types 13 Query "user": { "name": "Alan Cha" } }

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } { Response { user(user. Id: 1) { name } } Fields 14 Query "user": { "name": "Alan Cha" } }

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } { Response { user(user. Id: 1) { name } } Arguments 15 Query "user": { "name": "Alan Cha" } }

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } Return types 16 Query { Response { user(user. Id: 1) { name } } "user": { "name": "Alan Cha" } }

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } Query { { user(user. Id: 1) { name } } Compose fields into queries 17 Response "user": { "name": "Alan Cha" } }

Graph. QL basics Schema type Query { user(user. Id: Int!): User } type User { id: ID name: String } Query { Response { user(user. Id: 1) { name } } "user": { "name": "Alan Cha" } } Resolvers correspond to fields Resolvers produces values 18

Graph. QL basics Schema type Query { user(user. Id: number!): User } type User { id: ID name: String friends: [User] } 19 Query Response { { "user": { "name": "Alan Cha", “friends”: [ { name: “Louis Mandel” } { name: “Jim Laredo” } ] } user(user. Id: 1) { name friends { name } }

Response size Schema type Query { user(user. Id: number!): User } type User { id: ID name: String friends: [User] } x 3 20 Query { 1 User Response { "user": 2{Users "name": "Alan Cha", “friends”: [ { name: “Louis Mandel” } { name: “Jim Laredo” } ] } user(user. Id: 3 Users 1) { name friends { name } } } 1 User }

Number of resolvers Schema type Query { user(user. Id: number!): User } type User { id: ID name: String friends: [User] } x 3 21 Query { 1 user Response { "user": 1{ friends "name": "Alan Cha", “friends”: [ { name: “Louis Mandel” } { name: “Jim Laredo” } ] } user(user. Id: 1) { 1 friends name friends { name } } } 1 user }

Complexity measures Type complexity: size of the response Reflects the size of the data retrieved by a query Resolve complexity: number of resolvers Reflects the server’s query execution cost 22

Complexity formalization Type complexity Maximum list size or 1 if the field is not a list Type weight Resolve complexity Resolve weight 23 Maximum list size or 1 if the field is not a list Resolve complexity of sub-query Type complexity of sub-query

Complexity benefits Graph. QL service providers Graph. QL clients Interested in both complexities Interested in type complexity • • Load balancing Threat-prevention Resolver resource allocation Request pricing based on the response size or execution cost 24 Contracts with… • Graph. QL services • Network providers • Caching policies

Query Response None Schema type User { name: String friends: [User] } { } } Slicing Pagination type User { name: String friends(limit: Int!): [User] } { { Connection type User { name: String friends(first: Int, after: String): [User. Connection] } 25 friends: [ { … }, { … } ] friends(limit: 3) { name } } friends: [ { … }, { … } ] } { friends(first: 3) { nodes { name } } type User. Connection { total. Count: Int edges: [User. Edge] nodes: [User] } type User. Edge { cursor: String node: User } { friends { name } } { friends: { nodes: [ { … }, { … } ] } }

Configuration None Query type User { name: String friends: [User] } { Slicing Schema type User { name: String friends(limit: Int!): [User] } { Connection type User { name: String friends(first: Int, after: String): [User. Connection] } type User. Edge { cursor: String node: User } 26 friends { name } resolvers: “User. friends”: default. Limit: 10 resolver. Weight: 1 types: User: type. Weight: 1 friends(limit: 3) { name } resolvers: “User. friends”: limit. Arguments: [first] resolver. Weight: 1 types: User: type. Weight: 1 friends(first: 3) { nodes { name } } resolvers: “User. friends”: limit. Arguments: [first] limited. Fields: [edges, nodes] resolver. Weight: 1 types: User: type. Weight: 1 “/. +Connection$/”: type. Weight: 1 “/. +Edge$/”: type. Weight: 1 } } { type User. Connection { total. Count: Int edges: [User. Edge] nodes: [User] } } Configuration

Evaluation RQ 1: Can the analysis be applied to real-world Graph. QL APIs? RQ 2: Does the analysis produce cost upper-bounds? RQ 3: Are these bounds useful, i. e. close enough to the actual costs? RQ 4: Is our analysis cheap enough for use in API management? RQ 5: How does our approach compare to other solutions? 27

Query-response corpus • Nothing publicly available • Developed and open-sourced a Graph. QL query generator • Collected 10, 000 unique, anonymized query-response pairs from Git. Hub and Yelp Graph. QL APIs 28

Answering RQ 1 Can the analysis be applied to real-world Graph. QL APIs? Limit arguments: • Git. Hub and Yelp use consistent limit argument names first and last for Git. Hub, limit for Yelp configured with wild cards and regular expressions • Connection pattern in Git. Hub set using the limited. Fields property Default limits: • Required by 21 fields in Git. Hub and 13 fields in Yelp • Determined values using API documentation and experimental queries • API providers have these numbers readily available 29

Answering RQ 2 Does the analysis produce cost upper-bounds? 30

Answering RQ 3 Are these bounds useful, i. e. close enough to the actual costs? 31

Answering RQ 4 Is our analysis cheap enough for use in API management? 2017 Mac. Book Pro 8 -core Intel i 7 processor 16 GB of memory Median processing time 3. 0 ms for queries 1. 1 ms for responses 95% percentile 7. 3 ms for queries 4 ms for responses Linear time as a function of the query and response size 32

Answering RQ 5 How does our approach compare to other solutions? Closed-source Open-source Library A 33 Library B Library C

Answering RQ 5 How does our approach compare to other solutions? BQ 1: How large might the response be? BQ 2: How many resolver functions might be invoked? 34

Answering RQ 5 How does our approach compare to other solutions? 35

Answering RQ 5 How does our approach compare to other solutions? Library A Observations Overestimation and striations Problem: cannot define limit arguments Workaround: only use default limits 36

Answering RQ 5 How does our approach compare to other solutions? Library B and C Observations Underestimation Problem: cannot define default limits Workaround: none possible Overestimation Problem: cannot handle connection pattern Workaround: none possible 37

Answering RQ 5 How does our approach compare to other solutions? Two static analyses Static and dynamic analyses Can answer BQ 1 and BQ 2 with limitations Cannot answer BQ 1 or BQ 2 Static: Node limit analysis disregards connections pattern types Static: rejects queries based on levels of nesting Static: Call’s score analysis only counts resolvers that return Connection types Git. Hub’s metrics cannot be weighted Git. Hub’s focus on the connections pattern may be problematic 38 Dynamic: rate limits by executing queries and retroactively replenishing remaining quota Rejects valid queries that have high levels of nesting Possible to exceed Yelp’s rate limits

Contributions 1. Two distinct definitions of query complexity 2. Analysis can handle common schema conventions 3. Analysis built on formal Graph. QL semantics 4. First evaluation of such an analysis on real-world APIs 39

THANK YOU

Hypothetical REST API Get /users/alan_cha { first. Name: “Alan”, last. Name: “Cha”, friend. Ids: [“erik-wittern“, “gbaudart”, “james. D 123”], interests: [“graphql”, “programming”] } Get /users/erik-wittern { first. Name: “Erik”, last. Name: “Wittern”, friend. Ids: [“. . . ”], interests: [“. . . ”] } 41 Get /users/erik-wittern first. Name: “Erik”, last. Name: “Wittern”, friend. Ids: [“. . . ”], interests: [“. . . ”] }

Query Response None Schema type User { name: String friends: [User] } { } } Slicing Pagination type User { name: String friends(limit: Int!): [User] } { { Connection type User { name: String friends(first: Int, after: string): [User. Connection] } 42 friends: [ { … }, { … } ] friends(limit: 3) { name } } friends: [ { … }, { … } ] } { friends(first: 3) { nodes { name } } type User. Connection { total. Count: Int edges: [User. Edge] nodes: [User] } type User. Connection { cursor: string node: User } { friends { name } } { friends: { nodes: [ { … }, { … } ] } }

Query Response None Schema type User { name: String friends: [User] } { } } Slicing Pagination type User { name: String friends(limit: Int!): [User] } { { Connection type User { name: String friends(first: Int, after: string): [User. Connection] } 43 friends: [ { … }, { … } ] friends(limit: 3) { name } } friends: [ { … }, { … } ] } { friends(first: 3) { nodes { name } } type User. Connection { total. Count: Int edges: [User. Edge] nodes: [User] } type User. Edge { cursor: string node: User } { friends { name } } { friends: { nodes: [ { … }, { … } ] } }

Configuration None Query type User { name: String friends: [User] } { Slicing Schema type User { name: String friends(limit: Int!): [User] } { Connection type User { name: String friends(first: Int, after: string): [User. Connection] } type User. Edge { cursor: string node: User } 44 friends { name } resolvers: “User. friends”: default. Limit: 10 resolver. Weight: 1 types: User: type. Weight: 1 friends(limit: 3) { name } resolvers: “User. friends”: limit. Arguments: [first] resolver. Weight: 1 types: User: type. Weight: 1 friends(first: 3) { nodes { name } } resolvers: “User. friends”: limit. Arguments: [first] limited. Fields: [edges, nodes] resolver. Weight: 1 types: User: type. Weight: 1 } } { type User. Connection { total. Count: Int edges: [User. Edge] nodes: [User] } } Configuration

Complexity measures Type complexity Maximum list size or 1 if the field is not a list Type weight Resolve complexity Resolve weight 45 Maximum list size or 1 if the field is not a list Resolve complexity of sub-query Type complexity of sub-query

Query type complexity Configuration Schema Query resolvers: "Topic. related. Topics": limit. Arguments: [first] default. Limit: 10 resolver. Weight: 1 1 Topic 2 Topic(s) 0 scalar "Topic. stargazers": limit. Arguments: [first, last] limited. Fields: [edges, nodes] default. Limit: 10 resolver. Weight: 1 Stargazer: type. Weight: 1 0 scalar types: Topic: type. Weight: 1 1 Stargazer. Connection 0 scalar 2 Stargazer. Edge(s) 2 User(s) 0 scalar Total type complexity: 8 46

Query resolve complexity Configuration Schema Query resolvers: "Topic. related. Topics": limit. Arguments: [first] default. Limit: 10 resolver. Weight: 1 1 topic 1 related. Topics 0 trivial "Topic. stargazers": limit. Arguments: [first, last] limited. Fields: [edges, nodes] default. Limit: 10 resolver. Weight: 1 Stargazer: type. Weight: 1 0 trivial types: Topic: type. Weight: 1 1 stargazers 0 trivial 1 edges 2 nodes 0 trivial Total resolve complexity: 6 47

Response type complexity Configuration Schema Query resolvers: "Topic. related. Topics": limit. Arguments: [first] default. Limit: 10 resolver. Weight: 1 1 Topic 1 Stargazer. Connection 0 scalar 1 Stargazer. Edge 1 User 0 scalar types: Topic: type. Weight: 1 Stargazer: type. Weight: 1 1 Topic 0 scalar "Topic. stargazers": limit. Arguments: [first, last] limited. Fields: [edges, nodes] default. Limit: 10 resolver. Weight: 1 1 Topic 1 Stargazer. Edge Total type complexity: 8 48 1 User

Response resolve complexity Configuration Schema Query resolvers: "Topic. related. Topics": limit. Arguments: [first] default. Limit: 10 resolver. Weight: 1 1 topic 1 related. Topics 0 trivial "Topic. stargazers": limit. Arguments: [first, last] limited. Fields: [edges, nodes] default. Limit: 10 resolver. Weight: 1 1 stargazers 0 trivial 1 edges 1 nodes 0 trivial Stargazer: type. Weight: 1 0 trivial types: Topic: type. Weight: 1 Total resolve complexity: 6 49 1 nodes

Answering RQ 3 Are these bounds useful, i. e. close enough to the actual costs? query { organization (login: "nodejs") { repository (name: "node") { issues (first: 100) { nodes { repository { issues (first: 100) { nodes { repository {. . . }}}} 50

Answering RQ 5 How does our approach compare to other solutions? 51

Answering RQ 5 How does our approach compare to other solutions? 52

Discussion Configuration and applicability: • Our analysis is configurable to work with two real-world Graph. QL APIs • It is static and does not depend on any interaction with backend systems • Three strategies for managing over-estimation Unpaginated list field Tuning weights Hybrid static/dynamic system to leverage the graph data at runtime 53

Discussion Formalization and data-driven software engineering • Our formal analysis gives us provably correct bounds • We benefited from an understanding of Graph. QL Pagination Naming conventions • • • Incorporated pagination into our formalization Supported both of the widely used pagination patterns in our configuration Naming conventions inspired our support for regular expressions 54