GLSL Applications 1 of 2 Joseph Kider Source

GLSL Applications: 1 of 2 Joseph Kider Source: Patrick Cozzi – Spring 2011 University of Pennsylvania CIS 565 - Fall 2011

Agenda n GLSL Applications ¨ Per-Fragment Lighting ¨ Image Processing n Finish last week’s slides ¨ Open. GL Drawing ¨ Open. GL Multithreading

Per-Fragment Lighting Slide from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Per-Fragment Lighting Per-Vertex Lighting Per-Fragment Lighting Images from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Per-Fragment Lighting: Diffuse Slide from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Per-Fragment Lighting: Diffuse Slide from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Per-Fragment Lighting: Diffuse uniform vec 3 u_Color; in vec 3 fs_Incident; in vec 3 fs_Normal; in vec 3 fs_Texcoord; out vec 3 out_Color; void main(void) { vec 3 incident = normalize(fs_Incident); vec 3 normal = normalize(fs_Normal); float diffuse = max(0. 0 f, dot(-incident, normal)); out_Color = vec 4(diffuse * u_Color, 1. 0 f); }

Per-Fragment Lighting: Diffuse uniform vec 3 u_Color; in vec 3 fs_Incident; in vec 3 fs_Normal; in vec 3 fs_Texcoord; out vec 3 out_Color; • in vectors are not normalized. Why? • Good practice: don’t write to in variables void main(void) { vec 3 incident = normalize(fs_Incident); vec 3 normal = normalize(fs_Normal); float diffuse = max(0. 0 f, dot(-incident, normal)); out_Color = vec 4(diffuse * u_Color, 1. 0 f); }

Per-Fragment Lighting: Diffuse uniform vec 3 u_Color; • Know the graph: in vec 3 fs_Incident; in vec 3 fs_Normal; in vec 3 fs_Texcoord; out vec 3 out_Color; void main(void) • Why max? { vec 3 incident = normalize(fs_Incident); vec 3 normal = normalize(fs_Normal); float diffuse = max(0. 0 f, dot(-incident, normal)); out_Color = vec 4(diffuse * u_Color, 1. 0 f); } Graph from http: //www. wsd 1. org/waec/math/pre-calculus%20 advanced/Trigonometry/Graphing/graphingintro. htm

Per-Fragment Lighting: Specular Slide from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Per-Fragment Lighting: Specular Slide from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Per-Fragment Lighting: Specular uniform vec 3 u_Color; uniform vec 3 u_Spec. Color; uniform float u_Spec. Hardness; in in vec 3 fs_Incident; fs_Viewer; fs_Normal; fs_Texcoord; out vec 3 out_Color; void main(void) { vec 3 incident = normalize(fs_Incident); vec 3 normal = normalize(fs_Normal); vec 3 H = normalize(-incident + fs_Viewer); float specular = pow(max(0. 0 f, dot(H, normal)), u_Spec. Hardness); float diffuse = max(0. 0 f, dot(-incident, normal)); out_Color = vec 4(diffuse*u_Color + specular*u_Spec. Color, 1. 0 f); }

Per-Fragment Lighting: Specular uniform vec 3 u_Color; uniform vec 3 u_Spec. Color; uniform float u_Spec. Hardness; in in vec 3 fs_Incident; fs_Viewer; fs_Normal; fs_Texcoord; out vec 3 out_Color; “half” void main(void) { vec 3 incident = normalize(fs_Incident); vec 3 normal = normalize(fs_Normal); vector vec 3 H = normalize(-incident + fs_Viewer); float specular = pow(max(0. 0 f, dot(H, normal)), u_Spec. Hardness); float diffuse = max(0. 0 f, dot(-incident, normal)); out_Color = vec 4(diffuse*u_Color + specular*u_Spec. Color, 1. 0 f); }

Per-Fragment Lighting: Specular uniform vec 3 u_Color; uniform vec 3 u_Spec. Color; uniform float u_Spec. Hardness; in in vec 3 fs_Incident; fs_Viewer; fs_Normal; fs_Texcoord; out vec 3 out_Color; void main(void) { vec 3 incident = normalize(fs_Incident); vec 3 normal = normalize(fs_Normal); Blinn-Phong shading vec 3 H = normalize(-incident + fs_Viewer); float specular = pow(max(0. 0 f, dot(H, normal)), u_Spec. Hardness); float diffuse = max(0. 0 f, dot(-incident, normal)); out_Color = vec 4(diffuse*u_Color + specular*u_Spec. Color, 1. 0 f); }

Per-Fragment Lighting: Specular Slide from http: //www. opengl-redbook. com/SIGGRAPH/08/Open. GL. pdf

Image Processing n Our first look at GPGPU ¨ General-Purpose computation on Graphics Processing Units Input: Image n Output: Processed image n A kernel runs on each pixel n

Image Processing Examples Image Negative Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing Examples Edge Detection Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing Examples Toon Rendering Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing Questions Is the GPU a good fit for image processing? n Is image processing data-parallel? n What about bus traffic? n What type of shader should implement an image processing kernel? n

Image Processing: GPU Setup n Input: ¨ Texture ¨ Viewport-aligned n Output: framebuffer ¨ …for n quad, a. k. a full-screen quad now Kernel: fragment shader

Image Processing: GPU Setup 3. Each fragment shader can access any part of the image stored as a texel: gather 4. Each fragment shader executes the kernel and writes the color to the framebuffer Fragment shader 2. A fragment is invoked for each screen pixel 1. Render viewport-aligned quad Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing: GPU Setup 3. Each fragment shader can access any part of the image stored as a texel: gather 4. Each fragment shader executes the kernel and writes the color to the framebuffer Fragment shader 2. A fragment is invoked for each screen pixel 1. Render viewport-aligned quad Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing: GPU Setup 3. Each fragment shader can access any part of the image stored as a texel: gather 4. Each fragment shader executes the kernel and writes the color to the framebuffer Fragment shader 2. A fragment is invoked for each screen pixel 1. Render viewport-aligned quad Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing: GPU Setup 3. Each fragment shader can access any part of the image stored as a texel: gather 4. Each fragment shader executes the kernel and writes the color to the framebuffer Fragment shader 2. A fragment is invoked for each screen pixel 1. Render viewport-aligned quad Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing: GPU Setup 3. Each fragment shader can access any part of the image stored as a texel: gather 4. Each fragment shader executes the kernel and writes the color to the framebuffer Fragment shader 2. A fragment is invoked for each screen pixel 1. Render viewport-aligned quad Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing: GPU Setup n How do we model the viewport-aligned quad? Screen Two triangles? One big triangle?

Image Processing: GPU Setup n Which has more vertex shader overhead? ¨ Does it matter? Which is simpler to implement? n Which has less fragment shader overhead? n

Image Processing: GPU Setup n Triangle edges are redundantly shaded n Fragments are processed in 2 x 2 blocks ¨ Why?

Image Processing: GPU Setup n Triangle edges are redundantly shaded Fan Strip Max area FPS Number of vertices Image and Chart from http: //www. humus. name/index. php? page=News&ID=228

Image Processing: GPU Setup n The viewport has the same width and height as the image to be processed ¨ The texture also has the same dimensions ¨ How does the fragment shader access texels?

Image Processing: GPU Setup n Store texture coordinates per vertex Vertex Shader in vec 3 Position; in vec 2 Texcoords; out vec 2 fs_Texcoords; void main(void) { fs_Texcoords = Texcoords; gl_Position = vec 4(Position, 1. 0); } Fragment Shader uniform sampler 2 D u_Image; in vec 2 fs_Texcoords; out vec 4 out_Color; void main(void) { out_Color = texture(u_Image, fs_Texcoords); }

Image Processing: GPU Setup n Store texture coordinates per vertex ¨ What n memory costs does this incur? Does it matter? ¨ What bandwidth costs does this incur? ¨ What non-obvious optimization does it allow?

Image Processing: GPU Setup n Compute texture coordinate in fragment shader Vertex Shader in vec 3 Position; void main(void) { gl_Position = vec 4(Position, 1. 0); } Fragment Shader uniform sampler 2 D u_Image; uniform vec 2 u_inverse. Viewport. Dimensions; out vec 4 out_Color; void main(void) { vec 2 tx. Coord = u_inverse. Viewport. Dimensions * gl_Frag. Coord. xy; out_Color = texture(u_Image, tx. Coord); }

Image Processing: GPU Setup n Compute texture coordinate in fragment shader Vertex Shader in vec 3 Position; void main(void) { gl_Position = vec 4(Position, 1. 0); } What is u_inverse. Viewport. Dimensions? Fragment Shader uniform sampler 2 D u_Image; uniform vec 2 u_inverse. Viewport. Dimensions; out vec 4 out_Color; void main(void) { vec 2 tx. Coord = u_inverse. Viewport. Dimensions * gl_Frag. Coord. xy; out_Color = texture(u_Image, tx. Coord); }

Image Processing: GPU Setup n How do you access adjacent texels? uniform sampler 2 D u_Image; in vec 2 fs_Texcoords; out vec 4 out_Color; void main(void) { vec 4 c 0 = texture(u_Image, fs_Texcoords); vec 4 c 1 = texture. Offset(u_Image, fs_Texcoords, vec 4 c 2 = texture. Offset(u_Image, fs_Texcoords, vec 4 c 3 = texture. Offset(u_Image, fs_Texcoords, vec 4 = texture. Offset(u_Image, fs_Texcoords, out_Color = (c 0 + c 1 + c 2 + c 3 + c 4) * 0. 2; } ivec 2(-1, 0)); ivec 2( 0, -1)); ivec 2( 0, 1));

Image Processing: GPU Setup n How do you access adjacent texels? (0, 1) uniform sampler 2 D u_Image; in vec 2 fs_Texcoords; out vec 4 out_Color; void main(void) { vec 4 c 0 = texture(u_Image, fs_Texcoords); vec 4 c 1 = texture. Offset(u_Image, fs_Texcoords, vec 4 c 2 = texture. Offset(u_Image, fs_Texcoords, vec 4 c 3 = texture. Offset(u_Image, fs_Texcoords, vec 4 = texture. Offset(u_Image, fs_Texcoords, out_Color = (c 0 + c 1 + c 2 + c 3 + c 4) * 0. 2; } (-1, 0) (0, -1) ivec 2(-1, 0)); ivec 2( 0, -1)); ivec 2( 0, 1));

Image Processing: GPU Setup n texture. Offset requires constants ¨ E. g. ivec 2(x, y) is not allowed n How else do you access adjacent texels?

Image Processing: GPU Setup n How else do you access adjacent texels? uniform sampler 2 D u_Image; uniform vec 2 u_inverse. Viewport. Dimensions; out vec 4 out_Color; void main(void) { vec 2 tx. Coord = u_inverse. Viewport. Dimensions * gl_Frag. Coord. xy; vec 2 delta = 1. 0 / texture. Size(u_Image); vec 4 vec 4 c 0 c 1 c 2 c 3 c 4 = = = texture(u_Image, texture(u_Image, tx. Coord); tx. Coord + (delta out_Color = (c 0 + c 1 + c 2 + c 3 + c 4) * 0. 2; } * * vec 2(-1. 0, 0. 0))); vec 2( 0. 0, -1. 0))); vec 2( 0. 0, 1. 0)));

Image Processing: GPU Setup n How else do you access adjacent texels? uniform sampler 2 D u_Image; uniform vec 2 u_inverse. Viewport. Dimensions; out vec 4 out_Color; void main(void) { vec 2 tx. Coord = u_inverse. Viewport. Dimensions * gl_Frag. Coord. xy; vec 2 delta = 1. 0 / texture. Size(u_Image); vec 4 vec 4 c 0 c 1 c 2 c 3 c 4 = = = texture(u_Image, texture(u_Image, tx. Coord); tx. Coord + (delta out_Color = (c 0 + c 1 + c 2 + c 3 + c 4) * 0. 2; } * * vec 2(-1. 0, 0. 0))); vec 2( 0. 0, -1. 0))); vec 2( 0. 0, 1. 0)));

Image Processing: GPU Setup n How else do you access adjacent texels? uniform sampler 2 D u_Image; uniform vec 2 u_inverse. Viewport. Dimensions; (0, 1) out vec 4 out_Color; (-1, 0) void main(void) (0, -1) { vec 2 tx. Coord = u_inverse. Viewport. Dimensions * gl_Frag. Coord. xy; vec 2 delta = 1. 0 / texture. Size(u_Image); vec 4 vec 4 c 0 c 1 c 2 c 3 c 4 = = = texture(u_Image, texture(u_Image, tx. Coord); tx. Coord + (delta out_Color = (c 0 + c 1 + c 2 + c 3 + c 4) * 0. 2; } * * vec 2(-1. 0, 0. 0))); vec 2( 0. 0, -1. 0))); vec 2( 0. 0, 1. 0))); (1, 0)

Image Processing: Kernel Examples n Image Negative uniform sampler 2 D u_Image; in vec 2 fs_Texcoords; out vec 4 out_Color; void main(void) { out_Color = vec 4(1. 0) - texture(u_Image, fs_Texcoords); } Images from http: //web. engr. oregonstate. edu/~mjb/cs 519/Handouts/image. 1 pp. pdf

Image Processing: Kernel Examples n Gaussian Blur 2 D Gaussian Image from http: //www. librow. com/articles/article-9

Image Processing: Kernel Examples n Filter for 3 x 3 Gaussian Blur: [1 2 1] 1/16 * [2 4 2] [1 2 1] • The elements add to one • Other filters are also used for • Edge detection • Sharpen • Emboss • …

Image Processing: Kernel Examples n Gaussian Blur ¨ How would you implement the fragment shader? ¨ How is the memory coherence? n 3 x 3, 5 x 5, etc.

Image Processing: Kernel Examples Image from http: //www. ozone 3 d. net/tutorials/image_filtering_p 2. php

Image Processing: Kernel Examples n What does this filter do? [1 1 1] 1/9 * [1 1 1]

Image Processing: Read backs n How do we get the contents of the framebuffer into system memory? ¨ Print Screen? n It doesn’t matter if we are using: n Efficient read backs are important

Image Processing: Read backs n gl. Read. Pixels gl. Use. Program(/*. . . */); gl. Draw*(/*. . . */); unsigned char rgb = new unsigned char[width * height * 3]; gl. Read. Pixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, rgb); //. . . delete [] rgb;

Image Processing: Read backs n gl. Read. Pixels gl. Use. Program(/*. . . */); gl. Draw*(/*. . . */); Use GPU for image processing unsigned char rgb = new unsigned char[width * height * 3]; gl. Read. Pixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, rgb); //. . . delete [] rgb;

Image Processing: Read backs n gl. Read. Pixels gl. Use. Program(/*. . . */); gl. Draw*(/*. . . */); Allocate buffer for processed image unsigned char rgb = new unsigned char[width * height * 3]; gl. Read. Pixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, rgb); //. . . delete [] rgb;

Image Processing: Read backs n gl. Read. Pixels gl. Use. Program(/*. . . */); gl. Draw*(/*. . . */); unsigned char rgb = new unsigned char[width * height * 3]; gl. Read. Pixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, rgb); //. . . delete [] rgb; Ask for framebuffer’s color buffer

Image Processing: Read backs n gl. Read. Pixels gl. Use. Program(/*. . . */); gl. Draw*(/*. . . */); unsigned char rgb = new unsigned char[width * height * 3]; gl. Read. Pixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, rgb); //. . . delete [] rgb; You guys are sharp

Image Processing: Read backs n What is the major problem with gl. Read. Pixels?

Image Processing: Use Cases Photoshop-type applications n Post-processing in games n On the fly video manipulation n Augmented reality n These last three don’t even need read backs