Using DSLs in Practice Benefits and Challenges Using

Using DSLs in Practice - Benefits and Challenges Using DSLs in Practice www. mdsd-buch. de Benefits and Challenges Markus Völter voelter@acm. org www. voelter. de . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges About me • • Independent Consultant • Focus on • Software Architecture • Middleware • Model-Driven Software Development Markus Völter voelter@acm. org www. voelter. de Based out of Heidenheim, Germany . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Domain Driven Development • Domain Driven Development is about making software development more domain-related as opposed to computing related. It is also about making software development in a certain domain more efficient. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges A DSL and its context several Metamodel target software architecture design expertise subdomains composable multi-step multiple knowledge transform bounded area of knowlege/interest partial viewpoint Domain single-step compile Model semantics Ontology no roundtrip interpret precise/ executable Domain Specific Language graphical Metamodel textual . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges How DSLs are usually used • • Developer develops model(s) based on certain metamodel(s), expressed using a DSL. Using code generation templates, the model is transformed to executable code. • Alternative: Interpretation Optionally, the generated code is merged with manually written code. One or more model-to-model transformation steps may precede code generation. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges DSLs in Industry • Model-Driven Development (MDSD) pragmatic technology, process building blocks • OMG’s Model-Driven Architecture (MDA) standardization effort, technology-focus, platform independence, m 2 m transformations • Microsoft’s Software Factories (SF) framework for domain-specific IDE tooling, DSLs are part of this approach • Generative Programming (GP) traditional small scale, technology focused • Language-Oriented Programming (LOP) integrate DSLs into IDE with editors, debuggers, symbolic integration. ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Reasons for using DSLs • You want to provide a way for your domain-experts to formally specify their knowledge, and to provide a way for your technology people to define how this is implemented (using model transformations). • You might want to provide different implementations (i. e. more concrete models) for the same model, perhaps because you want to run it on different platforms (. NET, Java, CORBA). • You may want to capture knowledge about the domain, the technology, and their mapping in a clear, uncluttered format. • In general, you don’t want to bother with implementation details when specifying your functionality. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges MDSD Benefits I • Models are free of implementation artifacts – they directly represent domain knowledge and are thus reusable. • Implementations for various platforms can be generated in principle – the technology change problem is adressed to some extend. • Technology freaks and domain experts can take care of „their business“ (transformations and models, respectively) and need to care of each other‘s problems only in a limited way. • Domain experts can play a direct role in development since they can more easily understand models expressed with a DSL as opposed to implementation code. Domain Experts play the central role they deserve! . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges MDSD Benefits II • Development becomes more efficient since repetitive implementation code can be generated automatically. • Architectural contraints/rules/patterns can more easily be enforced, since they are embedded in the templates rather than just being documented (and ignored). This is especially important in really large teams, often in the context of Product-Line Engineering and Software System Families. • Transformer/Generator can address cross-cutting concerns (just like an aspect weaver) . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Benefits for Software Quality • DSLs, if they describe software structure/architecture requires an explicit, well-defined architecture. Defining an architecture this way improves the quality of the system (indpendent of whether it is generated or not). • Transformations capture expert knowledge. The generated code reflects this expert knowledge uniformly. • An DSL-based Archtitecture defines a strict programming model for the manually developed parts – again, uniformity and constrained-ness improves quality. • Generator does not produce accidental errors – either things are always right or always wrong. This makes finding errors easier! . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Benefits for Software Quality II • In general, defining a DSL forces you to take care of many good things, which you‘d like to have in any application development project: • Domain/Application Scoping • Variability Management • Well-Defined Software Architecture, Architecture Metamodelling • Trying to build a DSL/generator for a domain/target architecture enables your understanding of the domain/target architecture. This in itself is a huge benefit. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges DSL/Code Generation “Financial Benefits” • „Normal“ Implementation Effort . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges DSL/Code Generation “Financial Benefits” II • Realistic DSL based Implementation Effort . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges DSL/Code Generation “Financial Benefits” III • Ideal DSL based Implementation Effort . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges DSLs/Generation and Agility • Agile Manifesto: We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: - Individuals and interactions over processes and tools - Working software over comprehensive documentation - Customer collaboration over contract negotiation - Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. • DSL-based Models are no „paperwork“, they are the code which is translated to executable code automatically • Agility does not oppose tools in general – compilers are ok, model transformers are a kind of compiler . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges MDSD and Agility II • Project automation (ant, cruisecontrol) is ok in „agile minds“, so is automation of the writing of repetitive code • Automation of the development process makes responding to change easier and faster (single source principle). • Changes in the model respond to changes in the functional requirements • Changes in the templates/transformations can be used to evolve the architecture • The customer on-site can be integrated better, if we have languages that are better related to domain concepts as opposed to 3 GL code or the like. • Pair programming between developer and domain expert is more realistic. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges MDSD and Agility III • Tests can still be written manually (even before generation), generators can help is building mocks or scenarios • We have done Test-Driven, Model-Driven Development • We do not recommend a waterfall that first builds metamodel/DSL/generators and then builds apps, rather, both are iteratively evolved in parallel. • Domains Architectures are based on experience, not based on „big design upfront“ • Developers can do what they can do best: • Some deal with applications and customer requirements, • Others deal with technical architecture, platforms and generators • So: There is no conflict between Agility and DSLs/Generation!. ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Teaming issues • Using DSLs is not very different from “normal” programming – every developer can basically do it. • Defining DSLs is, however, something completely different: • Finding the „right“ abstractions, defining metamodels, keeping the various metalevels sorted, etc. is not everybody‘s business. • Some of the tools to define metamodels, DSLs, generators and model-2 -model transformations are not (yet) intuitively usable. • Therefore I recommend to keep DSL/generator development to a limited group of people in your project. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Teaming issues II - Roles • Not all of these roles are necessary in every project, of course • However, as an example, there is a fundamental difference between those who understand the domain and its abstractions (left branch) compared to those who know how to best use some platform technology (right branch) . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Iterative Dual Track Development • Develop DSL/Generator and at least one application at the same time. • Establish rapid feedback from application developers to domain architecture developers. • Develop both aspects iteratively and incrementally. Use strict timeboxing. • Infrastructure develops iteration n+1 whereas application developers use iteration n. • Introduce new DSL/Generator releases only at the beginning of iterations. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Iterative Dual Track Development II - Roles • Note that the „men“ in the diagram on the right are roles, you can easily have some of them be handled by the same person! . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Extract the Infrastructure • Before starting ITERATIVE DUAL-TRACK DEVELOPMENT, Extract the transformations from manually developed application. • Either, start by developing this prototype conventionally, then build up the DSL/Generator infrastructure based on this running application, • Or extract the code from applications developed in the respective domain before doing MDSD (but only if the quality is sufficiently good!) . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges How do I come up with a good metamodel? • Incrementally! • Based on experience from previous projects, and by „mining“ domain experts. • A very good idea is to start with a (typically) very well known domain: the target software architecture (platform) Architecture-Centric DSLs see below, Cascading . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges How do I come up with a good metamodel? II • In order to continuously improve and validate the metamodel for a domain, it has to be exercised with domain experts as well as by the development team. • In order to achieve this, it is a good idea to use it during discussions with stakeholders by formulating sentences using the concepts in the meta model. • As soon as you find that you cannot express something using sentences based on the meta model, • you have to reformulate the sentence • the sentence’s statement is just wrong • you have to update the meta model. • (Based on Eric Evans’ Ubiquitous Language) . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges How do I come up with a good metamodel? III • Example: • A component owns any number of ports. • Each port implements exactly one interface. • There are two kinds of ports: required ports and provided • • ports. A provided port provides the operations defined by its interface. A required port provides access to operations defined by its interface. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges One DSL is not enough • Most systems can be structured into various • partitions: functional subsystems • subdomains: technical aspects • It is hardly possible to describe each of these with the same DSL. • You will need to come up with separate DSLs • … that have to be „connectable“ in order to build the complete system . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges One DSL is not enough II • Structure your system into several technical subdomains such as persistence, GUI, deployment. • Each subdomain should have its own meta model and specifically, its own suitable DSL. • Define a small number of gateway metaclasses, i. e. meta model elements that occur in several meta models to help you join the different aspects together. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges One DSL is not enough III - Example . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Rich Domain-Specific Platform • Define a rich domain-specific application platform consisting of • Libraries • Frameworks • base classes • interpreters, etc. • The transformations will “generate code” for this domain-specific application platform. • As a consequence, the transformations become simpler. • DSLs and Frameworks are two sides of the same coin . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Rich Domain-Specific Platform II - Integration • A) Generated code can call non-generated code contained in libraries • B) A non-generated framework can call generated parts. • C) Factories can be used to „plug-in“ the generated building blocks • D) Generated classes can also subclass non-generated classes. • E) The base class can also contain abstract methods that it calls, they are implemented by the generated subclasses (template method pattern). ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Transformations as first class citizens • Transformations (and Templates) are central assets in MDSD. You should treat them accordingly. • Transformations should be versioned. • Refactor transformations to keep them current and well organized. • Modularize transformations, e. g. using object-oriented concepts such as encapsulation, polymorphism, inheritance, etc. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges External Model Markings (AO-Modelling) • In order to allow the transformation of a source model into a target model (or to generate code) it is sometimes necessary to provide “support” information that is specific to the target meta model. • Example: Entity Bean vs. Type Manager • Adding these to the source model “pollutes” the source model with concepts specific to the target model. • MDA proposes to add “model markings”, but this currently supported well by only very few tools. • Instead, we recommend keeping this information outside of the model (e. g. in an XML file); the transformation engine would use this auxiliary information when executing the transformations. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Testing through Model Verification • In many cases it is possible to detect design errors already in the models. This step is called model verification. • The most „extreme“ form is to interpret and simulate the whole model; this is however, not simple to achieve. • However, it is easily possible to verify design constraints in the model before model transformation or code generation steps are done. • So, a DSL definition includes checks („constraints“) that determine if a model is correct. • Note that those constraints report errors in a language that is on the same abstraction level as the DSL – i. e. no low level compiler errors! . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Testing through Model Verification II - Example • Example Metamodel . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Testing through Model Verification III – Example cont‘d • Verifications in the metamodel (Implemented) public class ECInterface extends generatorframework. meta. uml. Class { public String Check. Constraints() { Checks. assert. Empty( this, Attribute(), "must not have attributes. " ); } // more … } public class Component extends generatorframework. meta. Class { public String Check. Constraints() { Checks. assert. Empty( this, Operation(), "must not have attributes. " ); Checks. assert. Empty( this, Generalization(), "must not have superclasses or subclasses. " ); Checks. assert. Empty( this, Realization(), "must not implement any interface. " ); Checks. assert. Unique. Names( this, Port(), "a component's ports must have unique names. " ); } // more … }. ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Generation vs. Interpretation I • A model that can be used to generate code, can always also be interpreted. • The usual tradeoffs apply: Performance, Changeability at runtime • There are, however, additional things to consider: • When generating code, an explicit intermediate step (i. e. the generated code) is available. Might make things more easily understandable, since the structure is often well known (although this is not always true • Debugging is often easier, since you can debug „one metalevel down“ compared to an interpreter • In many cases, the platform (J 2 EE, Embedded Osek, etc. ) expects certain code structures. It is very hard to run an interpreter in these constraints. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Generation vs. Interpretation II • From my experience a combination of both approaches makes sense in the following way: • Code generation is used to generate structural artifacts (components, configuration files, documentation, etc. ). Components are artifacts that provide interfaces, require other interfaces, etc. • Different kinds of components use different metaphors for their implementation, such as • A rule engine component uses business rules to verify business transactions • A state chart based component manages persistent, long running processes • A mapping component is used to map data to legacy data structures (legacy system integration) • This implementation is often implemented as an interpreter. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Build an IDE • DSL-based Development often includes a wealth of input, intermediate and output artifacts. • Users (application developers) can easily be confused by these many artifacts. • To improve acceptance of the approach, you should consider building an IDE for application developers which • Hides unnecessary intermediate artifacts • Provides editors/support for “programming” with the DSL • Integrates all the validation, generation, build, etc. • Interactively connects and synchronizes the various models used • Eclipse, for example, is a good toolkit for building these kinds of IDEs. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Build an IDE: Example . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Adopting DSL-based Development . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Cascading DSLs • You would typically start with architecture-centric DSLs where the abstractions of the DSL correspond to the core concepts of the technical platform. • This automates many aspects of the technical development/infrastructure; • Results in a wide platform/infrastructure • Many projects can be handled with the infrastructure • In later phases, functional DSLs infrastructures will be built on this technical one, resulting in cascaded DSLs. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Cascading DSLs II . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Cascading DSLs III – M 2 M Transformations . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Cascading DSLs IV – M 2 M Transformations II . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Levels of MDSD V – M 2 M Transformations III • This approach works well in practice, if • you don‘t need to modify the intermediate transformation results manually • You can run Unit Tests in the generator to ensure the model-2 -model transformations are (and stay!) correct. ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Levels of MDSD VI – M 2 M Transformations IV . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges More or Less Open Issues • Composition of DSLs: There is not too much experience yet in compsing DSLs. • It works more or less, if you build all of them with the same tool (all UML-based, all ASCII-based) • Advanced IDE support for DSLs is limited, i. e. debuggers, context-aware editors, etc. • As long as you work „plain ASCII“ or UML based things work reasonably well – but this is often not an alternative • Integration with Versioning can be an issue, depending on the tool • Model-2 -Model Transformation languages are still not standardised (and maybe never will be, really). . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Summary • DSL-based software development can bring huge benefits to a development organization • Especially, if you work on product lines/software system families • To benefit, you have to make sure you adapt your processes and team structure accordingly • You also need to make sure the „rest“ of your processes work, i. e. customer integration, requirements manangement – DSLs cannot solve all your problems : -) • It is also important to select the right tools – the loudest marketing offensive does not necessarily lead to the best product. • Finally, be prepared to „improvise“ wrt. to some aspects of your development process. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges CONTENTS • • Introduction Benefits Challenges and Experiences • Process • Technical • Adoption • More or less Open Summary THE END. THANK YOU! . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r

Using DSLs in Practice - Benefits and Challenges Some advertisement • For those, who happen to speak (or rather, read) german: Völter, Stahl: Modellgetriebene Softwareentwicklung Technik, Engineering, Management d. Punkt Verlag, 2005 www. mdsd-buch. de • For all others: A translation (with Wiley) is under way. . ingenieurbüro für sof twaretechnologie w w w. vo © 2 0 0 5 M a rk u s V ö l t e r