Hyperledger Fabric Private Data Collection Types Types of

Hyperledger Fabric Private Data Collection Types

Types of collections Public data (via ordered blocks) Local In v 1. 2 plan Private Implicit definition All Idea to consider Immutable Membership Idea to consider Explicit def Idea to consider Mutable Membership* In v 1. 2 plan Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org Peer gossip distributes private data to all organizations Keep private data confidential from orderers Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. This is just normal chaincode data. Technically, there is no such thing as a ‘public collection’, it’s just public chaincode data, but including it here for comparison. Client Public Use Case A collection is a set of private data in a chaincode namespace. Who distributes private data to orgs? Peers Collection type Distribution *Note, another variation of Mutable Membership could create a collection dynamically (new API required) and distribute members as private data (in a local collection), and then going forward peers of these organizations use the private collection definition to distribute private data instead of the client doing it. A collection is defined by it’s data distribution policy (that is, collection membership, in all cases except for local collections), and has it’s own namespace within the chaincode. Key structure in state db: channel~chaincode~collection~key The different types of collections are based on different types of distribution (only). 2

Example - three orgs, plus a regulator Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Use Case Org. A Org. B Org. C In this example, a regulator gets all private data for audit purpose. Regulator Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org Peer gossip distributes private data to all organizations Keep private data confidential from orderers Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. AB R BC R AC R Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. AB R BC R AC R Private data distribution per transaction. private data – has access private data – no access, but has public hashes for validation Private data distribution per collection. Org. C gets all AC and BC data. 3

Collection definition – implicit vs explicit Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Use Case Org. A Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org Peer gossip distributes private data to all organizations Keep private data confidential from orderers Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. AB R BC R AC R There is an implicit collection for each unique combination of organizations in the channel. Note – This is only an idea for Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. AB R BC R AC R Explicitly specify named collections with specific MSPID memberships at chaincode instantiation time (aka definition). Memberships can change with each chaincode version. Only the last type of collection has an explicit named definition which needs to be created. There is one implicit ‘local’ collection per chaincode. In chaincode, specify empty string for collection name. There is one implicit ‘all’ collection per chaincode. In chaincode, specify ‘*’ for collection name. Note – This is only an idea for consideration at this point. Client specifies list of MSPIDs for distribution. consideration at this point. For the ledger collection namespace, use a hash of the MSPIDs. In chaincode, specify collection name. 4

Collection definition – org local state Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Use Case Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org Peer gossip distributes private data to all organizations Keep private data confidential from orderers Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. Org. A Note the special case: If a single MSPID is specified, this indicates local state for this org only. Useful for ‘org local state’ such as chaincode metadata package, org-specific approvals, thresholds, voting, etc. AB R BC R AC R A B C These are implict combinations of any one org, aka, ‘org local state. ’ There is an implicit collection for each unique combination of organizations in the channel. Client specifies list of MSPIDs for distribution. For the ledger collection namespace, use a hash of the MSPIDs. These are implicit combinations of any three orgs. These collections are implicit (does not need to be defined) and immutable (will never change). Note – This is only an idea for consideration at this point. In the short term, we would likely only have implicit support for the single org collections. 5

Collection namespaces Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Use Case Org. A Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org Peer gossip distributes private data to all organizations Keep private data confidential from orderers Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. AB R BC R AC R There is a namespace per implicit collection within the chaincode. Keys must be unique within each collection. This implies that collection members ‘own’ the namespace and can create any key. e. g. for the single MSPID collections, this is a reserved namespace for the org to write their keys. Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. AB R BC R AC R There is a namespace per explicit collection within the chaincode. Keys must be unique within each collection. This implies that collection members ‘own’ the namespace and can create any key. The implicit ‘local’ collection uses a single namespace. All keys must be unique. Facilitates asset transfer to any org in channel, while preserving provenance. The implicit ‘all’ collection uses a single namespace. All keys must be unique. Facilitates asset transfer to any org in channel, while preserving provenance. Example namespace: channel~chaincode~collection. ACR 6

State-based endorsement Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Use Case If we are worried about the extra perrecord storage space, this can be a reference to an endorsement policy. Org. A Use consistent state-based endorsement approach across all collection types, to simplify code and usage, and to faciliate simple asset transfer. Normal public data Endorsement policy can be stored in the key’s value as metadata. Facilitates asset transfer to any org in channel, while preserving provenance. Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org Endorsement policy can be stored in the key’s value as metadata. Facilitates asset transfer to any org in channel, while preserving provenance. Peer gossip distributes private data to all organizations Keep private data confidential from orderers Endorsement policy can be stored in the key’s value as metadata. Facilitates asset transfer to any org in channel, with provenance. Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. AB R BC R AC R Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. AB R BC R AC R Endorsement policy can stored in the key’s value as metadata. Facilitates asset transfer to orgs within same collection, while preserving provenance. Endorsement policy can be stored in the key’s value as metadata. Facilitates asset transfer to orgs within same collection, while preserving provenance. = keys that Org. A ‘owns’ via state-based endorsement In the case of private data, the endorsement policy would actually be stored in the public statedb within the value of the hash(key), so that non-members can also perform validation. 7

Collection – query implications Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Normal public data Use Case Org. A read-only queries Range query JSON query (Couch. DB) read-write txs Range query JSON query (Couch. DB) ✓ ✓ ✓ no phantom validation for couch queries ✓ ✓ phantom validation not possible no phantom validation for couch queries allowed on own org’s peer only? Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org allowed on own org’s peer only? Peer gossip distributes private data to all organizations Keep private data confidential from orderers ✓ ✓ ✓ no phantom validation for couch queries within collections within and across collections phantom validation not possible no phantom validation for couch queries Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. AB R BC R AC R Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. AB R BC R AC R allowed on own org’s peer only? within collections within and across collections One Couch. DB database with 3 collection namespaces. 8

Collection – query implications Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Org. A Use Case read-only queries read-write txs Range query JSON query (Couch. DB) ✓ ✓ ✓ no phantom validation for couch queries Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org ✓ ✓ phantom validation not possible no phantom validation for couch queries Peer gossip distributes private data to all organizations Keep private data confidential from orderers ✓ ✓ ✓ no phantom validation for couch queries Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between fixed set of orgs. Peer gossip distributes private data to all organizations in collection. Data privacy; Share private data between evolving groups of orgs. AB R BC R AC R within collections within and across collections phantom validation not possible no phantom validation for couch queries Due to collection namespace, can’t do key range queries across collections …unless we duplicate the state data with two key structures: collection~key and key~collection. Can be supported at data layer ‘for free’, requires additional shim API. With private data, peers that don’t have the data can’t validate the results Range queries re-executed at validation time to ensure no phantoms. No phantom validation for couchdb queries, some applications can tolerate phantoms, others can’t Range queries re-executed at validation time to ensure no phantoms. Note: only valid if we don’t allow missing data. 9

Collection – query implications Collection type Distribution Public data (via ordered blocks) Local Private Implicit definition All Immutable Membership Explicit def Mutable Membership Use Case Org. A read-only queries read-write txs Range query JSON query (Couch. DB) ✓ ✓ ✓ no phantom validation for couch queries Normal public data Client distributes private data to endorsing orgs, gossip shares within these orgs Collection membership, data, and owner privacy; asset transfer to any org ✓ ✓ phantom validation not possible no phantom validation for couch queries Peer gossip distributes private data to all organizations Keep private data confidential from orderers ✓ ✓ ✓ no phantom validation for couch queries Peer gossip distributes private data to all organizations in collection. within phantom How to handle? Data privacy; AB R collections and across validation Share private Need a chaincode config option: collections not possible BC R data between fixed set of allow. Read. Write. Transactions. Without. Phantom. Check orgs. AC R no phantom validation for couch queries Share private data between evolving groups of orgs. couch queries Peer gossip distributes private data to all organizations in collection. - true indicates to allow them (current behavior for Couch. DB JSON queries) within phantom no phantom if this is detected Data privacy; - false indicates to fail endorsement collections and across validation for AB R BC R AC R collections not possible 10

Deeper dive on state-based ownership, local collections, and asset transfer. 11

With state-based endorsement, there may be a different endorsement policy for each item in the collection. k 3 k 2 { BC k 1 { AB JSON { JSON } } AC ‘Endorsement policy’ gets added to value as metadata to support validation, similar to ‘version’. AB = Owned jointly by Org. A/Org. B BC = Owned jointly by Org. B/Org. C AC = Owned jointly by Org. A/Org. C Let’s simplify this by using a table view of the data… c 1 c 2 c 3 EP k 1 --- --- AB k 2 --- --- BC k 3 --- --- AC For k 1 with ownable state AB, client must send endorsement requests to Org. A and Org. B *Options exist for linkable as well as unlinkable ownership (e. g. ephemeral keys per state, Idemix endorsement credentials) 12

With local collections, data may be private to a subset of orgs, based on which orgs did the endorsement for a particular transaction c 1 c 2 c 3 EP k 1 --- --- AB k 3 --- --- AC c 1 c 2 c 3 EP k 1 --- --- AB k 2 --- --- BC c 1 c 2 c 3 EP k 2 --- --- BC k 3 --- --- AC Org. A Client For k 1 with ownable state AB, client must send endorsement requests to Org. A and Org. B Org. C 13

With local collections, data may be private to a subset of orgs, based on which orgs did the endorsement for a particular transaction Client from Org. A Clients are authorized to do non-key queries (range queries, couchdb json queries) against their own org only, results will include all private data that the org has access to. c 1 c 2 c 3 EP k 1 --- --- AB k 3 --- --- AC c 1 c 2 c 3 EP k 1 --- --- AB k 2 --- --- BC c 1 c 2 c 3 EP k 2 --- --- BC k 3 --- --- AC Org. A Org. B In this example, client from Org. A queries Org. A peer for all data that Org. A has access to. Note: non-key queries cannot be used in read/write transactions of private data, since it is impossible for other orgs to validate transactions for which they do not have the data. But client can do non-key query to understand state, and then propose key-based transactions. Org. C 14

Recall that with Side DB each org has private data (for endorsement) and public data (hashes for validation), so the local collection will actually look like this… private k 1 k 3 public c 1 c 2 c 3 EP H(k) H(v) EP --- --- AB H(k 1) H(v 1) AB H(k 2) H(v 2) BC H(k 3) H(v 3) AC --- --- AC Org. A Client private For k 1 with ownable state AB, client must send endorsement requests to Org. A and Org. B. public c 1 c 2 c 3 EP H(k) H(v) EP k 1 --- --- AB H(k 1) H(v 1) AB k 2 --- --- BC H(k 2) H(v 2) BC H(k 3) H(v 3) AC Transactions are validated on all peers, using the hashes, version numbers, and endorsement policies in the public state. private c 1 c 2 c 3 Org. B public EP H(k) H(v) EP H(k 1) H(v 1) AB k 2 --- --- BC H(k 2) H(v 2) BC k 3 --- --- AC H(k 3) H(v 3) AC Org. C 15

For change of EP, old EP and new EP must be fulfilled. (this is often equivalent to changing ownable state, e. g. asset transfer) AB private k 1 AC k 3 c 1 c 2 c 3 EP H(k) H(v) EP --- --- AB H(k 1) H(v 1) AB H(k 2) H(v 2) BC H(k 3) H(v 3) AC --- Client For k 1 transfer AB to AC, client must send endorsement requests to Org. A and Org. B and Org. C But wait, we have a problem! Org. C does not have the private data, and therefore cannot endorse transactions that read/write k 1… proceed to next slide… public --- AC private public c 1 c 2 c 3 EP H(k) H(v) EP k 1 --- --- AB H(k 1) H(v 1) AB k 2 --- --- BC H(k 2) H(v 2) BC H(k 3) H(v 3) AC private c 1 c 2 c 3 Org. A Org. B public EP H(k) H(v) EP H(k 1) H(v 1) AB k 2 --- --- BC H(k 2) H(v 2) BC k 3 --- --- AC H(k 3) H(v 3) AC Org. C 16

For change of EP, old EP and new EP must be fulfilled. (this is often equivalent to changing ownable state, e. g. asset transfer) AB private k 1 AC k 3 public c 1 c 2 c 3 EP H(k) H(v) EP --- --- AB H(k 1) H(v 1) AB H(k 2) H(v 2) BC H(k 3) H(v 3) AC --- Client --- AC private public c 1 c 2 c 3 EP H(k) H(v) EP k 1 --- --- AB H(k 1) H(v 1) AB k 2 --- --- BC H(k 2) H(v 2) BC H(k 3) H(v 3) AC Open question: Org. C will not have the prior history. Can Org. C now retrieve the prior history of k 1? How does it prove that it is entitled to it? Org. B /v 1 k 1 …not to worry… client can pass k 1 data to Org. C in a new transient field. As part of endorsement, peer will ensure hash of the passed k 1 matches h(k 1) in public state, and will use this value for any calls to Get. State(k 1), and then can successfully endorse the transfer. Org. A private c 1 c 2 c 3 public EP H(k) H(v) EP H(k 1) H(v 1) AB k 2 --- --- BC H(k 2) H(v 2) BC k 3 --- --- AC H(k 3) H(v 3) AC Org. C 17

Asset transferred AB private k 1 AC What did we accomplish? - Transferred a private asset, - with private verifiability via the hashes (likely a regulator/auditor would be involved to verify hashes against pre-images), - while maintaining queryability of data that each org has access to k 3 public c 1 c 2 c 3 EP H(k) H(v) EP --- --- AC H(k 1) H(v 1) AC H(k 2) H(v 2) BC H(k 3) H(v 3) AC --- --- AC private Org. B will no longer participate in endorsement and will not get future updates. public c 1 c 2 c 3 EP H(k) H(v) EP k 1 --- --- AC H(k 1) H(v 1) AC k 2 --- --- BC H(k 2) H(v 2) BC H(k 3) H(v 3) AC private Org. C is now an owner and must endorse future transactions. Org. A Org. B public c 1 c 2 c 3 EP H(k) H(v) EP k 1 --- --- AC H(k 1) H(v 1) AC k 2 --- --- BC H(k 2) H(v 2) BC k 3 --- --- AC H(k 3) H(v 3) AC Org. C 18

Asset transfer – Future AB k 1 AC A B C These are implict combinations of any one org, aka, ‘org local state. ’ Using the implicit collection for org-local state data, organizations will be able to set approval criteria. Organizations can set org-specific state that can serve as pre-approval instructions, used in chaincodes, or as conditions for endorsement. If proposal does not meet criteria, peer will not endorse. Examples: • pre-approve asset transfer from AB to AC. • pre-approve any trade if sale price is greater than some limit 19