Processing

Please wait...

Settings

Settings

Goto Application

1. WO2011007586 - METHODS AND SYSTEMS FOR REDUCING COMPRESSION ARTIFACTS

Note: Text based on automatic Optical Character Recognition processes. Please use the PDF version for legal matters

[ EN ]

DESCRIPTION

TITLE OF INVENTION: METHODS AND SYSTEMS FOR

REDUCING COMPRESSION ARTIFACTS

TECHNICAL FIELD

Embodiments of the present invention relate generally to image processing and, in particular, to methods and systems for reducing compression artifacts .

BACKGROUND ART

A digital image or a digital video sequence may be compressed by a compression algorithm in order to satisfy a bandwidth requirement of a transmission medium by which the image or sequence is to be communicated or to satisfy a storage-size limitation associated with a storage medium on which the image or sequence is to be stored. Exemplary, widely used, video-compression methods include MPEG- I , MPEG2 , H .264 and other video-compression methods. Many video compression methods are DCT (Discrete Cosine Transform) based and may represent a signal by the sum of smooth, oscillating waves. When bandwidth or storage is insufficient, -necessitating high compression ratios, decoded and displayed, or printed, images and videos may exhibit annoying ringing artifacts . Ringing artifacts may be caused by the fundamental inability of DCT-based compression algorithms to accurately represent edges, often making ringing artifacts more visible in the regions of text, diagrams, line drawings and other regions where strong, sharp edges are prominent. In video, some ringing artifacts may not always be visible. Ringing artifacts around stationary or slow-moving objects may be more visually obj ectionable than those near fast-moving obj ects. This may be due to the ability of the human visual system to better track objects with slow, or no motion, than obj ects with fast motion.

Some methods for de-ringing may be designed for use in a decoding loop or may assume knowledge of a decoding process . While such methods may take advantage of information in the codec (coder/ decoder) , these methods cannot be used in post-processing after decoding. Postprocessing after decoding may be necessary because, in practice, ringing artifacts that may not be visible in the output of the codec may become visible after signal processing, for example, enhancement, up-scaling and other signal processing, is applied to the decoded image or video .

Methods and systems for post-processing de-ringing may be desirable .

SUMMARY OF INVENTION

Some embodiments of the present invention comprise methods and systems for reducing artifacts in a digital image. In some embodiments of the present invention, a motion analyzer may determine a motion indicator associated with a pixel location in an image . An edge analyzer may determine an edge-strength indicator associated with the pixel location, and the motion indicator and the edge-strength indicator may be combined to effectuate filtering at the pixel location when the combination meets a first criterion.

The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

Fig. 1 is a chart showing exemplary embodiments of the present invention comprising filtering an image region based on a combination of edge analysis results and motion analysis results;

Fig. 2 is a chart showing exemplary embodiments of the present invention comprising edge-strength determination based on an average of gradient magnitudes in a local image region;

Fig. 3 is a chart showing exemplary embodiments of the present invention comprising filtering a pixel in an input image when the pixel is determined to be located in an area of the image likely to exhibit visually objectionable compression artifacts;

Fig. 4 is a chart showing exemplary embodiments of the present invention comprising filtering a pixel in an input image when the pixel is determined to be located in an area of the image likely to exhibit visually obj ectionable compression artifacts;

Fig. 5 is a chart showing exemplary embodiments of the present invention comprising determination of whether or not a pixel is located in an area of the image likely to exhibit visually obj ectionable compression artifacts and filtering the pixel according to the determination;

Fig. 6 is a chart showing exemplary embodiments of the present invention comprising determination of whether or not a pixel is located in an area of the image likely to exhibit visually obj ectionable compression artifacts and filtering the pixel according to the determination;

Fig. 7 is a chart showing exemplary embodiments of the present invention comprising pixel filtering using a switchable filter with multiple states, wherein the state of the switchable filter is selected based on a ringing-area indicator;

Fig. 8 is a chart showing exemplary embodiments of the present invention comprising image filtering based on a combination of an edge-strength indicator and a motion indicator; and

Fig. 9 is a system for use with the exemplary embodiments of the present invention.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The figures listed above are expressly incorporated as part of this detailed description.

It will be readily understood that the components of the present invention, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations . Thus, the following more detailed description of the embodiments of the methods and systems of the present invention is not intended to limit the scope of the invention, but it is merely representative of the presently preferred embodiments of the invention.

Elements of embodiments of the present invention may be embodied in hardware , firmware and/ or software . While exemplary embodiments revealed herein may only describe one of these forms, it is to be understood that one skilled in the art would be able to effectuate these elements in any of these forms while resting within the scope of the present invention.

Exemplary embodiments of the present invention may be described in relation to Figure 1 . In these exemplary embodiments, an input image 10 from a video sequence and temporally neighboring images 1 1 from the video sequence may be processed in a computing system. Edge analysis 12 may performed on the input image 10. Edge analysis 12 may comprise identification, in the input image 10 , of image areas near strong image edges. In some embodiments, edge analysis 12 may comprise calculation of an edge-strength indicator at each pixel in the input image 10. In alternative embodiments, edge analysis 12 may comprise, at each pixel in the input image 10, calculation of a local average of the edge strength values associated with the pixels in a neighborhood proximate to the pixel under consideration. Motion analysis 14 may be performed on the input image 10 and temporally neighboring images 1 1 . Motion analysis 14 may comprise identification, in the input image 10 , of image areas that are stationary or slowly changing with respect to the temporally neighboring images 1 1 . The results of the edge analysis 12 and the motion analysis 14 may be combined in ringing-area detection 16 to identify areas , in the input image 10, comprising both strong edges and stationary, or slow motion. The input image 10 may be filtered 18 by a smoothing filter, where the strength of the smoothing filter may be controlled by the outcome of the ringing-area detection 16, to produce an output image 20 that exhibits fewer visible artifacts .

In some embodiments of the present invention described in relation to Figure 1 and Figure 2 , edge analysis 12 may comprise identification of a strong image edge and identification of an image region substantially proximate to the strong image edge . In some embodiments, areas within approximately one transform-block size from a strong image edge may be identified, where the transform block size is the size of the transform used in the compression. A measure of edge strength at a pixel, which may be denoted EtJ for a pixel at location (z, /) in an image, may be computed by calculating 22 a gradient magnitude at the pixel. In some embodiments of the present invention, the gradient magnitude may be calculated using a Canny edge detector. In alternative embodiments of the present invention other edge detectors may be used. Exemplary edge detectors include a Roberts Cross edge detector, a Prewitt edge detector, a Sobel edge detector, a Rothwell edge detector, a Laplacian-of-Gaussian edge detector, a Laplace edge detector and other edge detectors known in the art. In some embodiments of the present invention, edge strength values may be averaged 24 in an analysis window centered at the pixel location (i,j) (target pixel location) to produce a local-region average edge strength at the center pixel location. The average edge strength, also considered the edge-strength indicator, associated with a pixel location (z,j) may be denoted ES1 . . In some embodiments of the present invention, the size of the analysis window may be related to the transform block size used in the compression. In alternative embodiments of the present invention, the size of the analysis window may be a preset default value . In some embodiments wherein the size of the analysis window is related to the transform block size used in the compression, the size of the analysis window may be approximately twice the transform block size . In some embodiments of the present invention where a variable block-size transform is used in the compression, the size of the analysis window may be related to the largest transform block size. In some embodiments of the present invention where the transform block size is unknown or unavailable, a default analysis window size of 8 (8 pixels by 8 pixels) may be used.

Transform size information may be obtained from the type of compression involved, which may be available on a system level, and knowledge of the video decoding process for a bitstream associated with the image sequence being post-processed is not required.

Motion analysis 14 may comprise identification of areas in the input image 10 where there is little or no change in relation to temporally neighboring images. In some embodiments of the present invention, motion analysis 14 may comprise motion estimation according to any of the well- known motion-estimation techniques known in the art. In these embodiments, a motion vector, which may be denoted mi j k - [_v,,j k>ui j,k] > may De estimated at each pixel location (z,/) (plurality of pixel locations) in the input image 10 associated with a temporal location, t, in the sequence, wherein u and v are components of the motion vector, k is the temporal index and T denotes transpose . The motion vector at a pixel location may measure the spatial displacement between the pixel location (i+vl J_k,j+ulJj!it + k) and the pixel location (ijj) , where the third index indicates the temporal relationship between the input image 10 and a temporally neighboring image . In some embodiments, a motion indicator, which may be denoted MI1 j , may be calculated according to M/(J = m( J J , where || -|| indicates a vector norm, for example, the L1 norm, the L2 norm, the LP norm, the L norm and any other vector norm. In some embodiments, multiple motion vectors may be available, and the motion indicator may be a combination of the multiple motion vectors . In some embodiments of the present invention, four motion vectors may be used to determine a motion indicator at a pixel. A motion indicator value at a pixel location quantifies the likelihood that the motion field at the pixel location is non-stationary. In some of these embodiments, motion vectors for two prior image frames and two later image frames may be used. In some embodiments of the present invention, the number and/ or

selection of neighboring image frames used in the determination of a motion indicator may be based on a tradeoff between resources and performance. In one exemplary embodiment, multiple motion vectors may be averaged according to:


where K is the number of motion vectors, to form the motion indicator. In alternative embodiments, a weighted average of the motion vectors may be used to form a motion indicator.

In some of these embodiments, the weight may be related to the temporal distance of a neighboring image from the input image 10.

In alternative embodiments of the present invention, motion analysis 14 may comprise motion detection according to any of the well-known motion-detection techniques known in the art. In these embodiments, a motion indicator, Ml1 n may have a value associated with the likelihood of motion determined by the motion detection. In some embodiments, a lower value may indicate a high likelihood of no , or little, motion.

Ringing-area detection 16 may combine the outcome of edge analysis 12 and motion analysis 14 to identify areas in the input image 10 that may be close to a strong edge and may be associated with little, or no, motion. That is, filtering is effected based on the combined motion indicator and the edge-strength indicator. In an exemplary embodiment, a binary indicator as a first criterion may be determined according to :

_ f 1 if M. j < thl & ESitJ > thl

Ij [θ otherwise

where R11 indicates that the location (i,j) may be in a ringing area when R1J =I and may not be in a ringing area when R1 J = Q , and thl and thl are predetermined threshold values . In alternative embodiments, a linear combination of the edge-strength indicator and the motion indicator may be used to determine a ringing-area indictor. In these embodiments, the ringing-area indicator may be formed according to:

Rt,j = aMtJ +bESiJ

wherein a and b are the weights in the linear combination. A person of ordinary skill in the art will recognize there are many ways of combining the edge-strength indicator and the motion indicator to form a ringing-area indicator.

A de-ringing filter may be applied 18 to the input image 10 to filter out ringing artifacts . Some embodiments of the present invention described in relation to Figure 3 may comprise a filter that is controlled by the ringing-area indicator. In these embodiments, a determination may be made 30 as to whether or not a pixel belongs to a ringing area. The determination may be based on the value of the ringing- area indicator, Rn , associated with the pixel. In some embodiments, the value of the ringing-area indictor may be compared to a threshold to determine if the pixel belongs to a ringing area (as a first criterion) . If the pixel does not belong 31 to a ringing area, then a determination may be made 32 as to whether or not there remain unfiltered pixels. If there are no unfiltered pixels remaining 33 , then the filtering process may terminate 34. If there are unfiltered pixels remaining 35, then the next pixel data may be obtained 36 and further examined 30.

If the pixel does belong 37 to a ringing area, then the pixel may be filtered 38. In some embodiments of the present invention, a two-dimensional, spatial-only filter may be used. In alternative embodiments, a three-dimensional, spatio-temporal filter may be used, for example, a three-dimensional bilateral filter or other spatio-temporal filter. In an exemplary embodiment, a two-dimensional bilateral filter may be used. In this exemplary embodiment, the filtered output at a pixel location (i,j) may be determined according to :


where l(i,j) denotes image intensity, m and n are the summation indices associated with the filter support and


where σl and σv denote the bilateral filter parameters and L is associated with the filter support. In an exemplary embodiment, σr = 15 , σs = 2 and L = 3 . After filtering 38, a determination may be made 32 as to whether or not there remain unfiltered pixels. If there are no unfiltered pixels remaining 33 , then the filtering process may terminate 34. If there are unfiltered pixels remaining 35 , then the next pixel data may be obtained 36 and further examined 30.

In some embodiments a three-dimensional bilateral filter can be used. In this exemplary embodiment, the filtered output at a pixel location (i,j) may be determined according to :

^T W(i, j, k, m, n,τ)l(i— m, j— n, k— τ)

-L≤m,n≤L


where l{i,j, k) denotes the image intensity of a frame at time k and

where σrj σs and σt denote the bilateral filter parameters, L is associated with the filter support, and Ti and T2 are associated with the temporal filter support.

In alternative embodiments described in relation to

Figure 4 , a determination may be made 40 as to whether or not a pixel belongs to a ringing area. The determination may be based on the value of the ringing-area indicator, RtJ , associated with the pixel. In some embodiments, the value of the ringing-area indictor may be compared to a threshold to determine if the pixel belongs to a ringing area. If the pixel does not belong 41 to a ringing area, then a determination may be made 42 as to whether or not there remain unfiltered pixels. If there are no unfiltered pixels remaining 43, then the filtering process may terminate 44. If there are unfiltered pixels remaining 45, then the next pixel data may be obtained 46 and further examined 40.

If the pixel does belong 47 to a ringing area, then filter parameters may be determined 48, and the pixel may be filtered 49. In some embodiments of the present invention, a two-dimensional, spatial-only filter may be used. In alternative embodiments, a three-dimensional, spatio-temporal filter may be used, for example, a three-dimensional bilateral filter or other spatio-temporal filter. In an exemplary embodiment, a two-dimensional bilateral filter may be used. In this exemplary embodiment, the filtered output at a pixel location (i,j) may be determined according to:


where I(i,j) denotes image intensity and


where L is associated with the filter support and
) and σ S (R, ,) denote the bilateral filter parameters which depend on R1 J and may be determined 48 for each pixel. In an exemplary embodiment, when Rt J indicates a ringing area, the parameters may be set according to σ,. (ringing) = 20 , σs (ringing) = 4 and L(ringing) = % , and when Rh] indicates a non-ringing area, the parameters may be set according to σr (non-ringing) = 10 , σs (non-ringing) = 2 and L (non-ringing) = 4 . In some embodiments, the filter parameters may be linearly scaled by R1 ; . In alternative embodiments, the filter parameters may be non-linearly scaled by RhJ . After filtering 49 , a determination may be made 42 as to whether or not there remain unfiltered pixels. If there are no unfiltered pixels remaining 43 , then the filtering process may terminate 44. If there are unfiltered pixels remaining 45, then the next pixel data may be obtained

46 and further examined 40.

In alternative embodiments, a first filter with a first default set of parameters may be used when a pixel is not

part of a ringing area, and a second filter with a second default set of parameters may be used when the pixel is part of a ringing area. In this case , the ringing area indicator acts as a switch (selecting) between a plurality of filters or filtering parameters. The second filter can also comprise the first filter associated with a second set of filter parameter values . In some embodiments, the second filter parameters may effectuate more aggressive smoothing. Some of these embodiments may be described in relation to Figure 5. In these embodiments, a determination may be made 50 as to whether or not a pixel belongs to a ringing area. The determination may be based on the value of the ringing-area indicator, R1 j , associated with the pixel. In some embodiments, the value of the ringing-area indictor may be compared to a threshold to determine if the pixel belongs to a ringing area. If the pixel does 51 belong to a ringing area, the pixel may be filtered with a first filter 52 , and if the pixel does not 53 belong to a ringing area, the pixel may be filtered with a second filter 54. After filtering 52 , 54 , a determination may be made 56 as to whether or not there remain unfiltered pixels. If there are no unfiltered pixels remaining 57, then the filtering process may terminate 58. If there are unfiltered pixels remaining 59 , then the next pixel data may be obtained

60 and further examined 50.

In some embodiments of the present invention, the first filter may comprise a two-dimensional, spatial-only filter. In alternative embodiments, a three-dimensional, spatio-temporal filter may be used for the first filter, for example, a three-dimensional bilateral filter or other spatio-temporal filter. In an exemplary embodiment, the first filter may comprise a two-dimensional bilateral filter. In this exemplary embodiment, the first-filter filtered output at a pixel location (i,j) may be determined according to:

]T W(iJ,m,n)[(i~mJ-n)

o(i i) = ~Ll≤m'n≤LX

-Ll≤m,n<L\

where l(i,j) denotes image intensity and


where σrl and σΛ denote the bilateral filter parameters associated with the first filter and Ll denotes the support of the first filter.

In some embodiments of the present invention, the second filter may comprise a two-dimensional, spatial-only filter. In alternative embodiments, a three-dimensional, spatio-temporal filter may be used for the second filter, for example, a three-dimensional bilateral filter or other spatio-temporal filter. In an exemplary embodiment, the second filter may comprise a two-dimensional bilateral filter. In this exemplary embodiment, the second-filter filtered output at a pixel location (z,y) may be determined according to :

∑ W(i,j,m,n)l(i~m,j-n)


where l(i,j) denotes image intensity and


where σr2 and σs2 denote the bilateral filter parameters associated with the second filter and Ll denotes the support of the second filter.

In yet alternative embodiments, a first filter with a first default set of parameters may be used when a pixel is not part of a ringing area, and a second filter with a second default set of parameters may be used when the pixel is part of a ringing area. In some embodiments, the first filter parameters and/ or the second filter parameters may be determined based on the ringing-area indicator. In some embodiments, the second filter parameters may effectuate more aggressive smoothing. Some of these embodiments may be described in relation to Figure 6. In these embodiments, a determination may be made 70 as to whether or not a pixel belongs to a ringing area. The determination may be based on the value of the ringing-area indicator, Rt J , associated with the pixel. In some embodiments, the value of the ringing-area indictor may be compared to a threshold to determine if the pixel belongs to a ringing area. If the pixel does 71 belong to a ringing area, filter parameters associated with a first filter may be determined 72 , and the pixel may be filtered with the first filter 74. If the pixel does not 75 belong to a ringing area, filter parameters associated a second filter may be determined 76, and the pixel may be filtered with the second filter 78. After filtering 74 , 78, a determination may be made 80 as to whether or not there remain unfiltered pixels . If there are no unfiltered pixels remaining 81 , then the filtering process may terminate 82. If there are unfiltered pixels remaining 83, then the next pixel data may be obtained 84 and further examined 70.

In some embodiments of the present invention, the first filter may comprise a two-dimensional, spatial-only filter. In alternative embodiments, a three-dimensional, spatio-temporal filter may be used for the first filter, for example, a three-dimensional bilateral filter or other spatio-temporal filter. In an exemplary embodiment, the first filter may comprise a two-dimensional bilateral filter. In this exemplary embodiment, the first-filter filtered output at a pixel location (i,j) may be determined according to :


where I(i,j) denotes image intensity and


where σrl (iζ 7) and σsl (Rt J ) denote the bilateral filter parameters associated with the first filter and Ll denotes the support of the first filter. In some embodiments, the filter parameters may be linearly scaled by R1 1 . In alternative embodiments, the filter parameters may be non-linearly scaled by R1 >; .

In some embodiments of the present invention, the second filter may comprise a two-dimensional, spatial-only filter. In alternative embodiments, a three-dimensional, spatio-temporal filter may be used for the second filter, for example, a three-dimensional bilateral filter or other spatio-temporal filter. In an exemplary embodiment, the second filter may comprise a two-dimensional bilateral filter. In this exemplary embodiment, the second-filter filtered output at a pixel location (z,y) may be determined according to:

∑ W(iJ,m,n)l(i~m,j-n)

Od Λ - -L2≤>*-»≤L2

(h Jh ∑ W(iJ,m,n)

-L2≤m,n≤L2

where I(i,j) denotes image intensity and

m,. . . . m2 + n2 I f (l(i, j)-l(i- m,j - n))2

where σr2 (i?, 7 ) and σi2 (l?, 7) denote the bilateral filter parameters associated with the second filter and Ll denotes the support of the first filter. In some embodiments, the filter parameters may be linearly scaled by Rl } . In alternative embodiments, the filter parameters may be non-linearly scaled by R} ) .

In still alternative embodiments of the present invention, pixel filtering may comprise a switchable filter with multiple states. These embodiments may be understood in relation to Figure 7. A determination may be made 90 as to whether or not there remain unfiltered pixels . If there are no unfiltered pixels 91 , then the filtering process may terminate 92. If there are unfiltered pixels 93 , then the data associated with the next pixel may be obtained 94. A ringing-area indicator associated with the pixel may be determined 95. A filter state may be selected 96 based on the value of the ringing-area indicator associated with the pixel. In some embodiments, a filter state may be associated with a range of ringing-area indicator values . The pixel may be filtered 98 according to a filter set to the selected state . In some exemplary embodiments, the filter state may be associated with the parameters of a two-dimensional bilateral filter. In these embodiments, the filter output at pixel location (i,j) may be

determined according to :

^ W(i,j,m,n)l(i~m,j -n)

-L≤m,n≤L

O(i,j) =

∑ W(i,j,m,n)

~L≤m,n≤L

where l(i,j) denotes image intensity and


where crrl (y) and <y {γ) denote the bilateral filter parameters associated with a state, denoted γ , where γ may map to an interval of R1 } values.

In some embodiments of the present invention comprising a bilateral filter of the form:

∑ W(i,j,m,n)l(i-m,j-n)

Oil A = -L≤m-n≤L

υ [''J ) ∑ W(i,j,m,n)

~L<m,n≤L

where l{i,j) denotes image intensity and


where σr and σs denote the bilateral filter parameters and L

denotes the filter support, the edge-strength measure may be calculated by summing the range kernel according to :


In these embodiments, computational expense may be reduced.

Some embodiments of the present invention may be described in relation to Figure 8. In these embodiments an input image (digital image) from a video sequence may be received 100 in a computing system. The edge strength values may be measured 102 for the input image. Temporally neighboring images may be received 104 , and motion indicators may be determined 106 for the input image and the temporally neighboring images. The edge strength values and the motion indicators may be combined 108 and the input image may be filtered 1 10 based on the combined edge strength values and motion indicators.

The above methods can also be used on a system for removing artifacts from a digital image, such as shown in Figure 9. In this case, the system includes a motion analyzer 1 14 for determining a motion indicator associated with a pixel location in the digital image ; an edge analyzer 1 12 for determining an edge-strength indicator associated with said pixel location in the digital image; and a combiner 1 16 for combining the motion indicator and the edge-strength indicator to effectuate filtering with a filter 1 18 at the pixel location, wherein the filtering at the pixel location is based on the combination.

The edge analyzer 1 12 can also include a range-kernel adder for summing a range kernel associated with the bilateral filter at the pixel location. The edge analyzer 1 12 can further comprise an edge detector for associating an edge-strength value with each pixel location in a first plurality of pixel locations proximate to the pixel location in the digital image; and an edge-strength-indicator associator for associating averaged edge-strength values with the pixel location.

The motion analyzer 1 14 can include a motion estimator for measuring, between said digital image and a temporally neighboring image, a spatial displacement associated with the pixel location. The motion analyzer 1 14 can further include a motion detector for detecting, between the digital image and a temporally neighboring image, a likelihood of motion associated with the pixel location.

The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding equivalence of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.