Responsive Designs in a Snap Nishant Sinha IBM

Responsive Designs in a Snap Nishant Sinha IBM Research, India Rezwana Karim Rutgers University, USA ESEC/FSE 2015 Proceedings of the 2015 10 th Joint Meeting on Foundations of Software Engineering 2016/06/30 M 2 ryosuke 1

Author • Nishant Sinha – IBM Reseach Labs, Research Staff Member • • Cognitive Systems Automated Bug Hunting Web Programming Formal Methods and Verification 2

Introduction • massive growth of mobile and tablet devices has compelled developers to create UIs viewable on multiple devices • Responsive design – optimal viewing experience across devices – requires deep expertise in HTML/CSS 3

Related Works • Adobe Reflow – assist users in adapting a particular design to different form-factors using direct manipulation – Designer must juggle through possible designs manually – Fixing designs is a laborious trial-and-error process • Grid-based CSS libraries (Bootstrap, etc. ) – allow designers to annotate UI elements with CSS classes having pre-defined responsive behaviors – hard to debug without a deep knowledge of the library and CSS layouts 4

DECOR • constraint-based design recommender tool, DECOR – assist the designer in creating multi-device responsive Uis – provides design recommendations to enable rapid multidevice design – Given an initial UI design, user-specified constraints on elements, e. g. , width or margin, and a list of device formfactors, DECOR provides ranked, device-specific recommendations to the designer for approval 5

DECOR (Example) Image navbar 2 newsbar content ip 4 navbar 1 ip 1 sidebar Constraints Min-width for input+button, navbar 1, ipi Hide side-bar Max-width for navbar 1 Constraints Max-width for navbar 1 6

DECOR • Design of DECOR relies on two main observations 1. UI designs are essentially labeled trees • responsive designs may be obtained by one or more tree transformations 2. Although the total number of potential tree transformations is huge, only a few transformation patterns are used in practice • • However, trying out all permutations on real UI design trees leads to a combinatorial explosion The authors devise several pruning heuristics to prune the search space 7

Outline • Design Tree • Constraint-Based Design Repair – Design Constraints – Design Fixes • Design Tree Graphs – Computing fix-chains – Optimization – Ranking Fix Sequences • Evaluation 8

Design Tree • Labeled tree used to represent UI designs Design Tree T=(N, ch, orn, σ) • Consists of set of nodes N • Maps ch, orn, σ • leaves of T correspond to basic UI widgets, e. g. , button, image, section, • intermediate nodes are containers ch(n) : the ordered list of children of n in T orn(n) : the layout of children of n σ(n): the styling, positioning and other attributes of n 9

Design Tree • Design Tree T=(N, ch, orn, σ) – orn(p)=(k, l) means that the children of p are aligned into a grid with k rows and l columns – orn(p)=(2, 3) p 10

Constraint-Based Design Repair • When transforming DT T (for bp) to DT T’ (for bp’) – We can obtain T ′ by shrinking the width of all elements, but this may lead to visually unpleasant design • To create a repair strategy, a “good-looking design” must be defined – It is almost impossible to quantify visual appeal precisely – design deficiencies can be characterized in terms of constraints on size, spacing and other attributes • E. g. an image is too wide, a paragraph text is too narrow 11

Constraint-Based Design Repair • If the designer can supply a set of design constraints on UI elements – a design satisfying these constraints is assumed to be visually appealing • Given a set of constraints C, T can be repaired in a large number of ways to satisfy C – designers rely on a small subset (about 10) of these fixes for creating responsive designs – This observation allows to devise a systematic approach to search over the set of possible fixes efficiently. 12

Design Constraints • logical predicates on properties of tree nodes – attributes (width, margin), orientation and the parent-child relationships – Example • for a paragraph UI element p, p. width ≤ 600 px is an instance of a max-width constraint 13

Design Fixes • Fix – basic unit of design tree transformation – Given a DT T=(N, ch, orn, σ) , a fix may modify either 1. 2. 3. 4. the set of nodes N (insert or delete nodes) the ordering of children ch the orientation orn one or more attributes (height, width, margin…) of a node n ∈ N 14

Design Fixes • Example – red-row fix at node n, orn(n) = (2, 2) (2 x 2 grid box) reduces the number of rows of n to orn’(n)= (1, 4) (1 x 4 grid box) 15

Design Fixes • In general, fixes and constraints interact in complex ways. applying a fix may lead to… – satisfaction of some constraints – violation of others • Fixes may enable or disable other fixes – Different permutations of fixes may lead to different designs – Exploration of several permutations of multiple fixes is needed – Authors introduce design tree graphs to explore representative fix sequences systematically 16

Design Tree Graphs • Design Tree Graph G=(Q, ι, ∑, R, Ψ, T) over a fix library F and constraints C – – – Q : set of states ι : set of initial states ∑ : set of transition labels R : transition relation T (・): mapping function, where T(q) is the DT T corresponding to state q – Ψ : Ψ ⊆ Q consists of good states q with satisfying trees 17

Design Tree Graphs • Design Tree Graph G=(Q, ι, ∑, R, Ψ, T) over a fix library F and constraints C • Finding responsive designs boils down to exploring paths of G to find a good state 18

Design Tree Graphs • Algorithm for computing all satisfying fix-chains – This algorithm is inefficient, optimized later in the paper 19

Computing fix-chains • Suppose after shrinking, the width of imgi in T 0 violates C • In this case, bad nodes (all nodes whose some descendant violates C) in T 0 are {imgi, Vi, H 1} • Suppose EXPLOREDTG in Alg. 1 picks a bad node V 1 to fix, and • applies red-margin to V 1 (still imgi is too narrow) • apply fixes red-row (shrinks imgi further), followed by red-col to V 1 • at this point, it detects a cycle and backtracks without success • after trying to fix V 2, V 3 and V 4 similarly, the algorithm backtracks to the initial state q 1 without success • Now EXPLOREDTG picks node H 1 and applies fixes • Algorithm will find fix-chains (H 1 , red-margin)→(H 1 , red-col), and a shorter chain (H 1, red-col) 20

Computing fix-chains • EXPLOREDTG is inefficient in multiple ways 1. The algorithm tries to fix Vi’s unnecessarily • red-row for Vi should be avoided because applying red-row will not satisfy the min-width constraint 2. The algorithm may try to both fix Vi followed by H 1 and vice versa. In most cases, fixing nodes in the tree order (H 1 followed by Vi) is sufficient • Given a tree T with N nodes and fix library F , the number of fix-chains in the corresponding DTG is O(N! · p. N) where p = (| F | +1) 21

Optimizing DTG Exploration • path-pruning heuristics to explore a small representative set of fix-chains in DTG based on 1. pruning the fixes applied to each node 2. selecting the order of nodes to fix • The latter may omit some satisfying DTs from exploration; but no desired DTs were eliminated in the evaluation 22

Valid and Enabled Fixes • valid fixes – fixes that can be applied because their pre-condition is satisfied • invalid fixes – fixes that cannot be applied because their pre-condition is not satisfied – E. g. We can’t apply red-margin on a node with no margin • GETVALIDNODEFIXES – Computes only valid fixes and prunes away the rest • GETVALIDFIXES – Combines these fixes for the bad nodes in T and ranks them to obtain enabled fixes 23

Redundant fix pruning • prune the valid fix set by removing fixes which may not help satisfy any violated constraint in C – If n is a bad node but none of descendants are bad, then fixes red-col and red-row are redundant – Applying red-col (red-row) on n is redundant if none of its descendants is violating min-width (max-width) constrain – Applying red-margin if 1. max-width is violated for n 2. n violates its min-width constraint and maximum margin reduction is not enough 24

Detecting equivalent fix-chains • Independent Fixes – Two fixes f 1 and f 2 on nodes n 1 and n 2, respectively, are said to be independent if 1. applying one fix does not disable the other 2. T ′ obtained after applying both f 1 and f 2 does not depend on the order in which the fixes are applied • The notion of independence allows us to explore only the representative fix-chains of enabled fixes – Most fixes commute with each other, but depending on the given constraints, fixes may disable each other 25

Dependent fixes • Suppose in the initial state q 1, the min-width criteria for both B 2 and B 3 is violated 1. applying fix f 1 (red-col) on B 1 2. applying fix f 2 (red-margin) on B 2 and B 3 • Two fixes are dependent on each other because applying one fix disables the other 26

Pre-order Fix Chains • Fixes on higher tree nodes affect a large number of nodes and few such fixes (in contrast to lower node fixes) are sufficient to yield most desired DTGs • Pre-order fix chains – fixes are applied on the parent node first, followed by children – process siblings in a fixed order • Given a tree T with N bad nodes, number of subsequences of pre(T) is O(N · p!) where p =|F| + 1 – significantly lower than the earlier bound O(N ! · p. N ) 27

Ranking Fix Sequences • DECOR ranks the obtained fix chains as follows – Shorter fix-chains are ranked higher than longer ones – For fix-chain s and s′ of equal length, s is ranked higher if between nodes ni and n′i at ith index, ni has a higher tree height than n′i 28

Implementation & Evaluation • DECOR is implemented as a plugin to Maqetta, an open source WYSIWYG editor – plugin extracts the design as a JSON file and sends it to a backend server that provides responsive design suggestion 29

Benchmarks • No standardized set of benchmarks for responsive designs exist – Authors selected web-sites which implement the popular responsive patterns 1. Synthetic designs – few boxes, each of which demonstrating one or two responsive behaviors 2. Real-world designs – selected 7 web-sites having multiple responsive patterns and created mockups in Maqetta 30

Evaluatioin • Environment – Mac Snow Leopard with 2. 3 GHz Intel Core i 5 processor and 4 GB memory • Implemented 3 exploration modes in DÉCOR – Dynamic Depth limit (DDL) • once DECOR finds a fix-chain of length l, it restricts its search to all fix chains of depth l. – Bounded exploration • Given a particular bound l, DECOR restricts its search to all fix-chains of length less than l in DTG – Lazy mode • DECOR stops after finding single fix-chain, which is minimized by greedily removing fixes and checking if the remaining chain repairs the tree. This mode aims at finding a single, short fix-chain quickly. 31

Results 32

Effects of optimization 3 hours 30 seconds RFP: redundant fix pruning preorder: searching only for pre-order fix chains 33

Effects of optimization the correlation of enabled fixes and run-time for overview benchmark 34

DDL vs Lazy mode 35

Conclusion • presented a systematic approach for inferring multidevice responsive designs from a given UI design and user-specified constraints • DÉCOR – aims at improving the productivity of experienced designers for creating responsive designs as well as reducing the learning curve for HTML/CSS and responsive designs for unskilled designers – able to provide realistic list of recommendations in a short time for real-life designs 36