Electrical Engineering National Central University PrePostProcessing VideoAudio Processing

Electrical Engineering National Central University Pre/Post-Processing 蔡茗光 Video-Audio Processing

Electrical Engineering National Central University Outline • Pre/Post-Processing Overview • Pre-Processing introduction • Post-Processing introduction • System Block diagram Video-Audio Processing

Electrical Engineering National Central University Pre/Post-Processing Overview Generally, the pre/post-processing is like the following︰ Input Pre. Processing Encoder Post. Processing Goal︰the former enhance coding efficiency by removing noise information without compromising quality, the latter reduce the blocking(Grid Noise)、ringing(Staircase Noise) effect Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(1/X) input Color conversion Down conversion Prefiltering output • Basically it can be separated into three stages shown in the above — Color conversion images are transformed in a more convenient form ex︰RGB HSL、RGB YUV Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(2/X) — Down conversion images are down sampled for data reduction ex︰ 422 420、422 411 — Pre-filtering reduce the complexity of video sequences by attenuating noise and small figures ( that is smoothing ), the resulting frames are less prone to errors such as blocking、ringing 、temporal flicker. it can be divided into three portions frequency domain temporal domain spatial domain Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(3/X) Examples of noise - Random Noise source Residual noise Film blotch and scratch noise Compression artifacts ……. - Impulse Noise source Satellite glitches Analog clamping errors Bit errors in digital transmission ……. Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(4/X) ◆ Frequency domain ( in the same frame ) Transform input data to frequency domain(ex︰DFT、DCT. . ) g(t) = h(t) * f(t) G(w) = H(w) F(w) A Butter-worth LPF is illustrated below (1 D - form )︰ | H(w) |2 = Video-Audio Processing 111111 1+(w / wp)2 n H w wp n frequency response input frequency pass-band frequency order

Electrical Engineering National Central University Pre-Processing Introduction(5/X) ◆ Frequency domain ( in the same frame ) g(x, y) = h(x, y) * f(x, y) G(u, v) = H(u, v) F(u, v) A Butter-worth LPF is illustrated below ( 2 D - form )︰ | H(u, v) | = 1111 1+[ D(u, v) / D 0 ]2 n H(u, v) frequency response D(u, v) input frequency D 0 cut-off frequency n order Two variables (D 0 、n) can be tuned when implementing. Generally, n should be small to avoid ringing Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(6/X) Video-Audio Processing Original n=4 Wp = 7 n=1 Wp = 7 n=4 Wp = 10

Electrical Engineering National Central University Pre-Processing Introduction(7/X) ◆ Temporal domain ( in the different frame ) Linear︰ the following is a de-interlaced vertical temporal filter neighboring field(s) current field weighted sum interpolated pixel Video-Audio Processing original pixel

Electrical Engineering National Central University Pre-Processing Introduction(8/X) ◆ Temporal domain ( in the different frame ) Non-Linear︰ the following is a de-interlaced vertical median filter Previous field Current field interpolated pixel which is median result of three arrows original pixel Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(9/X) Original Verticalmedian Squaremedian Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(10/X) Original Framemedian MBmedian Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(11/X) ◆ Spatial domain ( in the same frame ) Linear︰ w 1 w 2 w 3 P 1 P 2 P 3 1 1 w 4 SUM w 7 w 5 w 6 P 4 P 5 P 6 w 8 w 9 P 7 P 8 P 9 pi original pixel value Pi new pixel value wi weighting ( integer ) 1 1 9 i=1 P 5 = Sum(piwi) SUM Video-Audio Processing SUM = 9 i=1 Sum(w i)

Electrical Engineering National Central University Pre-Processing Introduction(12/X) ◆ Spatial domain ( in the same frame ) Non-linear (ex︰median、max、min、average)︰ p 1 p 2 p 3 P 1 P 2 P 3 p 4 p 5 p 6 P 4 P 5 P 6 p 7 p 8 p 9 P 7 P 8 P 9 pi original pixel value Pi new pixel value P 5 = Video-Audio Processing 9 i=1 median(p i)

Electrical Engineering National Central University Pre-Processing Introduction(13/X) Original Framebased MBbased Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(14/X) I-frame P-frame Video-Audio Processing

Electrical Engineering National Central University Pre-Processing Introduction(15/X) From the table, a problem is generated in the MB-based filter. bit rate is higher than the original frame PSNR is lower than the original frame The reason may be About bit rate︰ due to the noise variance in the same MB, the median value would be different About PSNR︰ due to the uncontinuous edge, it’ll make the situation more serious Video-Audio Processing

Electrical Engineering National Central University Post-Processing Introduction(1/X) • Commonly, it can be partitioned into two parts shown below input De. Blocking De. Ringing output — De-Blocking、De-Ringing reduce the artifacts due to the quantization of the DCT coefficients, the degradation mainly consists of two kinds of artifacts︰ Video-Audio Processing

Electrical Engineering National Central University Post-Processing Introduction(2/X) 1. the gradual intensity changes in original image become abrupt intensity variations along block boundaries ( Grid Noise ), 2. while the pixel values at either side of an edge is modified, increasing the degradation of the entire edge ( Staircase Noise ) Video-Audio Processing

Electrical Engineering National Central University Post-Processing Introduction(3/X) For areas near block edge a low-pass filtering is performed by ultilizing fuzzy computation of its coefficients a block area near edge fine detailed area For fine detailed areas filtering isn’t applied Video-Audio Processing

Electrical Engineering National Central University System Block Diagram(1/X) Video in T Q Q-1 T-1 MC/ME Video-Audio Processing Loop filter

Electrical Engineering National Central University System Block Diagram(2/X) Video in T Q Q-1 T-1 Loop filter Video-Audio Processing MC/ME

Electrical Engineering National Central University System Block Diagram(3/X) filter T Q Q-1 Video in T-1 Loop filter Video-Audio Processing MC/ME

Electrical Engineering National Central University A Question Video-Audio Processing