BROCKMANN CONSULT Programming with BEAM Norman Fomferra Carsten

BROCKMANN CONSULT Programming with BEAM Norman Fomferra Carsten Brockmann 20. – 22. 01. 2009 Finnish Environment Institute, Syke Helsinki

BROCKMANN CONSULT Day 1: Introduction 1. Software Overview An introduction to VISAT, its user interface elements and the BEAM command line tools. Includes a detailed presentation of the BEAM standard I/O format. 2. Architecture and APIs A high level introduction into the BEAM 4. x architecture including the diverse application programming interfaces. 3. Runtime Configuration A detailed presentation of the BEAM installation directory structure, the standard modules, the 3 rd party libraries used, the main entry points into the BEAM code. 4. Development and Build Process An introduction to Java development with BEAM. The goal is to compile and run a simple program that uses the BEAM API with your preferred Java IDE. Programming BEAM, Syke, 20. -22. 01. 2010 2

BROCKMANN CONSULT Day 2: Using the BEAM API 1. A simple Data Processor We develop a simple data processor, which will ingest an AATSR Level 1 data product and write a trivial cloud mask. The processor will use a simple command line interface. We will then review the output in VISAT. 2. BEAM Graph Processing Framework (GPF) A detailed introduction to the new BEAM graph processing framework. 3. Developing a GPF Operator We convert the simple data processor into a GPF operator. The processor will use the standard command line interface provided by the GPF. 4. BEAM Scripting An introduction to the new Scripting API introduced in Java 1. 6 and a discussion of its relevance for BEAM. Programming BEAM, Syke, 20. -22. 01. 2010 3

BROCKMANN CONSULT Day 3: Extending BEAM 1. BEAM Module Development An introduction to BEAM modules (plug-ins), the module descriptor file and module activators. 2. The Maven Build System Maven is a highly configurable build system which allows us to quickly create a module for BEAM. 3. Creating a BEAM Module We add a module descriptor to the code developed so far in order to create a versioned, deployable and updateable BEAM module. The GPF operator becomes part of BEAM. 4. Integration with VISAT We develop a simple graphical user interface for the GPF operator. We integrate the GPF operator and its GUI into VISAT by adding a command which allows the users to invoke the operator from the VISAT Tools menu. Programming BEAM, Syke, 20. -22. 01. 2010 4

BROCKMANN CONSULT Wish list of participants • Writing plug-ins (like a simple algorithm) • Batch processing • Automatic retrieval of new satellite products (wouldn't that be a nice feature) • Command-line interfacing, interfacing with MATLAB • Automatically producing graphics or georeferenced sub-products • Pixel value retrieval, coordinate-based retrieval for a large sample set • Overlaying transect data from external sources with beam-supported products Programming BEAM, Syke, 20. -22. 01. 2010 5

BROCKMANN CONSULT Why develop with BEAM? • • Access EO data, metadata, no-data, flags Simple & effective programming models Reuse a rich software infrastructure It is free, it is open source, its Java Review and reuse source code BEAM code is widely used, it is in use BEAM is supported by ESA Programming BEAM, ESRIN, 22. -26. 09. 2008 6

BROCKMANN CONSULT Development Use Cases 1. Use it § Write batch-mode scripts § Program stand-alone applications § Develop web-services 2. Extend it § Add EO data readers, writers § Add EO data processing nodes § Add map projections, DEMs § Add VISAT actions, tool bars and tool windows 3. Clone it § Customize and brand VISAT Programming BEAM, ESRIN, 22. -26. 09. 2008 7

BROCKMANN CONSULT Architecture Overview Programming BEAM, ESRIN, 20. -22. 01. 2010 8

BROCKMANN CONSULT Module-based Architecture • BEAM module: – Name – Description – Version – Authors – Changelog – Dependencies – Extension points – Extensions • Simple extension model: – Host module provides extension point, e. g. “product-reader” – Client module provides extension to host's extension point, e. g. “seawifs-reader” – Clients can be hosts Programming BEAM, ESRIN, 20. -22. 01. 2010 9

BROCKMANN CONSULT VISAT Module Manager Programming BEAM, Syke, 20. -22. 01. 2010 10

BROCKMANN CONSULT Development Platform Usages Programming BEAM, Syke, 20. -22. 01. 2010 11

BROCKMANN CONSULT GPF - Graph Processing Framework • Rapid processor development § Forget about processing environment § Concentrate on data and algorithm Programming BEAM, ESRIN, 22. -26. 09. 2008 12

BROCKMANN CONSULT BEAM Development Process at BC • 3 -4 core developers • 1 -2 application developers • 1 permanent tester • Developers also do user support • Agile Development Process • Feature list planning with ESA every 2 months • Weekly iteration meetings • Daily stand-up meetings Programming BEAM, Syke, 20. -22. 01. 2010 13

BROCKMANN CONSULT Tools used for BEAM Development • Development § Intelli. J IDEA (10) and Eclipse (2) IDEs § Version control system is Subversion § Maven - Project model & dependency management • Testing § JUnit – achieve good tested(10) code coverage § QF Test – Automatic application & GUI testing • Build § Maven – The Java make § Install 4 j – Installer builder § Team. City build server Programming BEAM, Syke, 20. -22. 01. 2010 14

BROCKMANN CONSULT Getting started • Online programming tutorial and guides www. brockmann-consult. de/beam/wiki • From BEAM website § Download & install BEAM 4. 6. 1 § Download BEAM API documentation § Download BEAM source code • Install a Java IDE § Intelli. J IDEA, Eclipse, Net. Beans Programming BEAM, ESRIN, 22. -26. 09. 2008 15

BROCKMANN CONSULT Naming Conventions in Java Code • Interface / Class names § Camale. Case, first character upper-case § Geo. Coding, Product, Raster. Data. Node • Variable names § Camel. Case, first character lower-case § background. Color, threshold, min. Value • Constant names § All letters upper-case, underscore § DIALOG_TITLE, BUFFER_SIZE_LIMIT • Method / Function names § Camel. Case, first character lower-case § get. Sample. Value, compute. Tile , is. Closed • Package names § all lower-case § beam, syke, binning Programming BEAM, Syke, 20. -22. 01. 2010 16

BROCKMANN CONSULT BEAM Installation Directory • beam-4. x. y § bin BEAM installation directory Application binary and script files - visat. exe, gpt. bat, - ceres-launcher. jar, . . . § config Application configuration file(s) - beam. config § lib Common, 3 rd -party libraries - xtream. jar, jdom. jar, - gt-*. jar, . . . § modules Application and plug-in BEAM modules - ceres-*. jar - beam-*. jar Programming BEAM, Syke, 20. -22. 01. 2010 17

BROCKMANN CONSULT Invocation of Command-line Tools Invocation of gpt. bat via a dedicated launcher program (cereslauncher. jar). All JARs found in BEAM’s lib and modules directories are put on the classpath: @echo off set BEAM 4_HOME=C: Program Filesbeam-4. 6. 1 "%BEAM 4_HOME%jrebinjava. exe" ^ -Xmx 1024 M ^ -Dceres. context=beam ^ "-Dbeam. main. Class=org. esa. beam. framework. gpf. main. Main" ^ "-Dbeam. home=%BEAM 4_HOME%" ^ "-Dncsa. hdflib. HDFLibrary. hdflib=%BEAM 4_HOME%. . . jhdf. dll" ^ "-Dncsa. hdf 5 lib. H 5. hdf 5 lib=%BEAM 4_HOME%. . . jhdf 5. dll" ^ -jar "%BEAM 4_HOME%binceres-launcher. jar" %* exit /B 0 Programming BEAM, Syke, 20. -22. 01. 2010 18

BROCKMANN CONSULT The BEAM Graph Processing Framework (GPF) Programming BEAM, Syke, 20. -22. 01. 2010

BROCKMANN CONSULT Terms • ((EO) Data) Processing System § A system that utilizes hardware and manages processes running on it • ((EO) Data) Processor § A program that transforms source products into target products • ((EO) Data) Processing Framework § A software and API with an architecture designed for developing processors • Framework § “Don’t call us, we call you. ” § Abstract interfaces comprising callback functions clients must implement Programming BEAM, Syke, 20. -22. 01. 2010 20

BROCKMANN CONSULT Processing System (1) • Main objective: Translate a job into a number of tasks running in parallel on a cluster of machines • Scalability – Add CPUs – Add more storage space • Data locality – Avoid network I/O – Force local data I/O • Load balancing – Assign tasks to idle compute nodes • Backup tasks – Handle tasks taking unusually long time to complete • Fault tolerance – Recover from HW failures • Monitoring – Statistics & Performance 21

BROCKMANN CONSULT Processing System (2) • A processing system places some requirements on the processes (processors) it is running: § A processor shall be configurable - Input and output files - Processing parameters - Runtime behaviour (e. g. memory, tiling, buffer sizes) § A processor shall be observable - Progress Status Logging Exit code Programming BEAM, Syke, 20. -22. 01. 2010 22

BROCKMANN CONSULT A Processing Graph Programming BEAM, Syke, 20. -22. 01. 2010 23

BROCKMANN CONSULT Processing Framework Requirements • Create processing chains or even graphs composed of processing nodes. Processing nodes shall § be reusable in other graphs, e. g. MERIS Cloud Screening § be easily configurable in terms of processing parameters § be easily configurable in terms of the algorithm implementation • Processor implementation code shall abstract from § physical file format of EO data § file I/O § configuration & parameterisation, parameter value access • Avoid I/O overhead between processing steps • Exploit multi-core CPU architectures • The framework shall be extendible, e. g. it shall recognise new processing node implementations at runtime Programming BEAM, Syke, 20. -22. 01. 2010 24

BROCKMANN CONSULT BEAM GPF • Allows to construct directed, acyclic processing graphs • Implements the “pull-processing” paradigm § A processor is represented by its processing graph § Product processing requests are translated into raster-tilerequests propagated backwards through the graph § Tile requests are automatically parallelised depending on CPU cores • Simple data processing and programming model § For a particular node, clients (Java developers) implement how a raster tile is being computed § Parameter values are “injected” by the framework • Generated user interfaces: Command-line (gpt) and VISAT GUI • Operators can be dynamically added to BEAM via plug-in modules Programming BEAM, Syke, 20. -22. 01. 2010 25

BROCKMANN CONSULT Product Data Model (1) Programming BEAM, ESRIN, 22. -26. 09. 2008 26

BROCKMANN CONSULT Product Data Model (2) Programming BEAM, ESRIN, 22. -26. 09. 2008 27

BROCKMANN CONSULT GPF Architecture (1) Programming BEAM, ESRIN, 22. -26. 09. 2008 28

BROCKMANN CONSULT GPF Architecture (2) • Clients provide their compiled Operator Java code packed into JAR modules • A JAR module publishes its operator services via metadata (Service Provider Interface, SPI) § META-INF/services/ org. esa. beam. framework. gpf. Operator. Spi • Once GPF is started, e. g. by visat. exe or gpt. bat, it loads all operator services it finds on its (dynamic) classpath § %BEAM_HOME%/modules/*. jar § %BEAM_HOME%/lib/*. jar Programming BEAM, Syke, 20. -22. 01. 2010 29

BROCKMANN CONSULT GPF Processing Node Anatomy (1) @Operator. Metadata(alias = “Noise. Red. Op", version = "1. 0", authors = "Ralf Quast", copyright = "(c) 2008 by Brockmann Consult", description = "Performs a noise reduction on” + “ CHRIS/Proba images. ") public class Noise. Red. Op extends Operator { @Source. Product(alias = "source") private Product source. Product; @Target. Product private Product target. Product; @Parameter(default. Value=“false”) private boolean slit. Correction; @Parameter(interval=“(0, 50]”, default. Value=“ 25”) private int smoothing. Order; @Parameter(value. Set={“N 2”, “N 4”, “N 8”}, default. Value=“N 2”) private String neighbourhood. Type; //. . . } Programming BEAM, Syke, 20. -22. 01. 2010 30

BROCKMANN CONSULT GPF Processing Node Anatomy (2) If single target bands can be computed independently of each other, e. g. NDVI or algorithms which perform single band filtering: public class Noise. Red. Op extends Operator { public void initialize() throws Operator. Exception { // validate and process source products and parameter values // create, configure and set the target product } public void compute. Tile(Band target. Band, Tile target. Tile, Progress. Monitor pm) throws Operator. Exception { // Obtain source tiles for used bands of source products // Process samples of source tiles to samples of target tile // Set samples of single target tile } } Programming BEAM, Syke, 20. -22. 01. 2010 31

BROCKMANN CONSULT GPF Processing Node Anatomy (3) If single target bands cannot be computed independently of each other, e. g. model inversion algorithms based on neural networks: public class Noise. Red. Op extends Operator { public void initialize() throws Operator. Exception { // validate and process source products and parameter values // create, configure and set the target product } public void compute. Tile. Stack(Map<Band, Tile> target. Tiles, Rectangle target. Tile. Rectangle, Progress. Monitor pm) throws Operator. Exception { // Obtain source tiles for used bands of source products // Process samples of source tiles to samples of target tiles // Set samples of all given target tiles } } Programming BEAM, Syke, 20. -22. 01. 2010 32

BROCKMANN CONSULT How GPF “drives” Processing Nodes 1. GPF knows about all registered Operators (Plug-in) 2. GPF encounters a required node (GUI, CLI, XML graph) 1. The client‘s Operator class is found by its alias 2. The Source. Product, Target. Product and Parameter annotations are analysed 3. The client‘s Operator object is created 4. Source products and parameter values are injected 3. GPF calls Operator. initialize(). In this method, the client provides code to 1. validate and process source products and parameter values 2. create, configure and set the target product 4. GPF calls either Operator. compute. Tile() or Operator. compute. Tile. Stack()in case raster data is required Programming BEAM, ESRIN, 22. -26. 09. 2008 33

BROCKMANN CONSULT gpt – The GPF Command-Line Tool Programming BEAM, Syke, 20. -22. 01. 2010 34

BROCKMANN CONSULT GPF Processing Node Configuration Programming BEAM, Syke, 20. -22. 01. 2010 35

BROCKMANN CONSULT GPF Processing Node GUI Programming BEAM, Syke, 20. -22. 01. 2010 36

BROCKMANN CONSULT GPF Programming Model Product p 0 = Product. IO. read. Product(“MER_1 P. N 1”); Map<String, Object> params 1 =. . . ; Map<String, Object> params 2 =. . . ; Map<String, Object> params 3 =. . . ; Product p 1, p 2, p 3; p 1 = GPF. create. Product(“Smile. Corr. Op”, params 1, p 0); p 2 = GPF. create. Product(“Cloud. Mask. Op”, params 2, p 1); p 3 = GPF. create. Product(“Noise. Red. Op”, params 3, p 1, p 2); Write. Op. write. Product(p 3, new File(“TEST. DIM”), “BEAM-DIMAP”, Progress. Monitor. NULL); Programming BEAM, Syke, 20. -22. 01. 2010 37

BROCKMANN CONSULT End Thanks for using BEAM! http: //www. brockmann-consult. de/cms/web/beam/forum Programming BEAM, Syke, 20. -22. 01. 2010