Audio and Video Communication in Multiplayer Games through

Audio and Video Communication in Multiplayer Games through Generic Networking Middleware Maarten Wijnants Wim Lamotte

Outline p p Introduction Network. Intelligence Middleware n n p p Application and Network Awareness Generic but Extensible Integration in Existing NVE Application Experimental Results Conclusions & Future Work Questions

Introduction p p p Advancements in many different areas of game design and development However: still little support for the real-time streaming of multimedia content in on-line multiplayer games Nonetheless has a number of interesting applications n n In-game audiovisual communication Dynamic streaming of multimedia content based on user location/interest, … (e. g. video advertisements)

Introduction p Audio and video communication n p Easier to input and less awkward than textual chat Inherently transfers emotional state of players Attaches a distinct voice and face to nicknames Enables players to communicate in a natural and immersive way Current on-line multiplayer games n n n Still support textual chat Some also support voice communication Virtually no support for video communication

Introduction p The lack of support for multimedia streaming in games can be attributed to n The high bandwidth requirements of these streams p p n Games should be playable on both p p Game developers need to implement it efficiently Client network bandwidth is a scarce resource low-bandwidth dial-in modems high-bandwidth DSL or cable modem connections requires extra code to be written We present our networking middleware that facilitates inclusion of multimedia streaming

Network. Intelligence Middleware p Our networking middleware consists of a number of interconnected proxy servers n n n Proxies are positioned at edge of the network Clients need to connect to a proxy Proxy is responsible for p p p forwarding relevant network streams to the client routing packets sent by client to their destination Each proxy is network aware n n n Has knowledge of the transportation network Proxy periodically probes the network links of connected clients throughput, delay, … Proxy records bandwidth usage of streams

Network. Intelligence Middleware p Each proxy is also application aware n n Has high-level knowledge of the application Provided by the client application p n Developers can utilize a provided Network. Intelligence layer to take care of the client/proxy communication p p p n E. g. client could keep proxy up-to-date regarding the relative importance of the different multimedia streams that are currently used in the application Only needs to be interfaced with the client’s awareness manager Automatically responds to probing packets Continuously queries the awareness manager Differentiates our middleware from other proxy systems!

Network. Intelligence Middleware p Based on its compound awareness, the middleware intelligently manages the bandwidth of connected clients n E. g. players move around in the virtual world p p p n Relative importance of individual streams changes Proxy can detect this due to its application awareness Changes allocation of client's bandwidth accordingly E. g. middleware can intelligently react to fluctuations in a client's downstream capacity p p Proxy can detect this due to its network awareness When bandwidth drops, the middleware determines which streams should be reduced in quality or blocked client bandwidth no longer exceeded

Network. Intelligence Middleware p Middleware should be applicable in a wide range of networked applications n n We therefore opted for a generic design However, many applications require specific middleware functionality p p Proxy plug-in mechanism ensures extensibility Application developers can implement own plug-in E. g. video transcoding plug-in It should cost developers minimal effort to integrate our middleware in their application n n Implementing a plug-in = deriving from a base class and adding desired functionality NILayer prevents that the client application needs to be completely recoded

Integration in Existing NVE App p We integrated our middleware in our in-house developed NVE framework n n n Supports video-based avatars Video streams need to be exchanged between users in real-time Supports voice communication Closely resembles a multiplayer game

Integration in Existing NVE App p Some technical details of the framework n The awareness manager of the NVE is region-based p p p n Each region has a MC address for distributing events AM determines which regions this client should be aware of Only the multicast groups of these regions are joined Uses 3 distinct video qualities p p p For reasons of scalability (see table) Each region has 3 distinct video multicast addresses Video awareness manager selects the video quality High Quality (HQ) Medium Quality (MQ) Low Quality (LQ) Codec H. 263 Resolution CIF (352 x 288) FPS 25 15 15 Bitrate (bps) 200. 000 100. 000 50. 000 n Separate multicast group per region for audio streams

Integration in Existing NVE App p Actual integration of the middleware in the NVE proved to be straightforward. It sufficed to: n n p Insert the NILayer in the client software Link the NILayer with the different awareness managers The NILayer n n n Extracts information about the currently selected regions Monitors the virtual distance between the local user and the other clients Informs the proxy of relative importance of the different multimedia streams, based on this positional information More priority is given to streams coming from nearby clients This is where the middleware gets its application awareness

Experimental Results p Experiment n 3 unmodified NVE clients: C 1, C 2 and C 3 p p n 2 clients connected through our middleware: PA and PB p n n No audio or video These 5 clients were located in the same region of the virtual world (see figure for exact client positioning) Client PA p p n Video Audio Fixed downstream bandwidth (320 Kbps) Moved around in the virtual world Client PB p p Fluctuating bandwidth: 440 ->320 ->600 Kbps Remained stationary in the virtual world

Experimental Results p Multimedia network packets received by client PA:

Experimental Results p Multimedia network packets received by client PB:

Conclusions p What does this experiment learn us? n The current downstream capacity of a client's network connection is always respected p n The middleware exploits application awareness to intelligently distribute a client's downstream capacity over the different multimedia streams p n This is due to the middleware's network awareness Stream playback at client-side will normally improve Less important streams are reduced in quality or blocked before more significant streams are altered Clients receive the streams that are most important to them at an as high as possible quality Developers need not spend time on these issues Our middleware facilitates the integration of efficient realtime streaming functionality in networked applications

Future Work p p The networking middleware presented here is a work in progress In the future we would like to n n Add device awareness to the middleware Develop some additional plug-ins for our networking middleware p n E. g. a plug-in which adds 3 D audio mixing functionality Integrate our work into a “shared workspace” application p p Goal is to efficiently support real-time meetings between both collocated and distributed team members Audio and video communication will play an important role

Questions?

Additional Results p C 1 stops transmitting audio C 2 video boosted