Applying Differentiable Programming to the Dark Channel Prior

Applying Differentiable Programming to the Dark Channel Prior Vandy Tombs July 24, 2020 Julia. Con 2020 ORNL is managed by UT-Battelle, LLC for the US Department of Energy

Differentiable Programming • Goal: Combine Machine Learning with validated algorithms and physical models – Interpretability – Requires less data – Improve science-based models with data • Apply gradient based optimization to these algorithms – Parameterize the algorithm – Define a loss function – Gather data 1 2 Open slide master to edit

Hazy Imagery • Modeling Hazy images • I (x): hazy image • J (x): haze-free image • A: global atmospheric light • is the transmission map Hazy Image Clear Image 2 3 Open slide master to edit

The Dark Channel Prior • Prior assumption: Clear images have one color channel with pixel intensity close to zero within an image patch – Estimating the transmission map • Refine the transmission map » Gaussian and Soft matting » Guided Filter – Estimating Atmospheric Light • The top 0. 1% of the brightest pixels in the dark channel are selected • Then the pixel with the highest intensity value is considered atmospheric light 3 4 Open slide master to edit

DCP Implementation and Zygote. jl 1 function dehaze(img: : Abstract. Array{T}; 2 darkchannel: : Dark. Channel = original_darkchannel, 3 atmosphere_percent = T(0. 2), select = intensity, 4 smooth: : Function, D = nothing, t 0 = T(0. 1), 5 rho = T(0. 0), omega = T(. 95)) where {T<: Abstract. Float} 6 dark = darkchannel(img) 7 a = atmospheric_light(img, dark, select, atmosphere_percent, D) 8 A = reshape(a, (1, 1, : )) 9 t = 1. - omega. * darkchannel(img. / A). + rho 10 smoothed_t = apply_smooth(smooth, img, t) 11 return (img. - A). / max. (smoothed_t, t 0). + A 12 end Zygote. jl Considerations: – Learnable parameters must be inputs to the function – Mutating arrays Dark Channel Atmospheric Light Transmission Map Refinement Inverse Haze Model 3 5 Open slide master to edit

Parameterizing the Dark Channel Construction • Use either a min filter, a mean filter or a combination of both • Learn the Dark Channel construction • Replace with a small neural network 1 mutable struct He. Dark. Channel{T} <: Dark. Channel 2 min_vs_mean: : T 3 filter_size: : NTuple{2, Int} 4 function He. Dark. Channel(m: : T, f) where {T} 5 0 ≤ m ≤ 1 || error("min_vs_mean must be 6 between 0 and 1. ") 7 new{T}(m, f) 8 end 9 end Dark Channel Atmospheric Light 1 struct Learn. Dark. Channel <: Dark. Channel 2 model 3 end Transmission Map Refinement Inverse Haze Model CNN 4 6 Open slide master to edit

Using Julia's Types • Define an abstract type for each parameter family • Define a concrete type for each parameter • Benefits: – Custom methods to update the gradient in a unique way – Each parameter could use a different optimizer for applying the updates – Custom gradients – Simplifies hyperparameter optimization implementation 3 7 Open slide master to edit

Parameterizing the Transmission Map • Dark Channel CNN Atmospheric Light Transmission Map Refinement Inverse Haze Model Gaussian Filter Guided Filter CNN 8 5 Open slide master to edit

A Generalization of Sequential Forward Selection • Dark Channel CNN Atmospheric Light Transmission Map Refinement Inverse Haze Model Gaussian Filter Guided Filter CNN 9 6 Open slide master to edit

Results: Initialization Dark Channel Atmospheric Light Transmission Map Refinement Inverse Haze Model No Filter Hazy Dehazed 7 10 Open slide master to edit

Results: Learned Model Dark Channel Atmospheric Light Transmission Map Refinement Inverse Haze Model No Filter Hazy Dehazed 8 11 Open slide master to edit

Conclusion: Differentiable Programming in Julia • Benefits – Implement the Dark Channel Prior (DCP) without too much concern with how Zygote. jl or differentiable programming would be used – Simple to apply differentiable programming to the DCP implementation – Parallelization in Julia improved training time • Difficulties – Zygote. jl does not easily work with mutating arrays – Learned parameters must be inputs to the functions being differentiated 9 12 Open slide master to edit

Thank You! 13 Open slide master to edit