3 Flow Control in Compressed Video Communications 3.1 Introduction In multimedia communications, compressed video streams need to be transmitted over networks that have inconsistent and time-varying bandwidth requirements. To make the best use of available network resources at any time and guarantee a maximum level of perceptual video quality from the end-user’s perspective, a certain flow control mechanism must be introduced into the video communication system (Cote et al., 1998; Wang, 2000). Over-rating the output of a video coder can cause an undesirable traffic explosion and lead to congested networks. On the other hand, uncontrolled reduction of the output bit rate of a video coder leads to unnecessary quality degradation and inefficient use of available bandwidth re- sources. Flow control techniques must then be employed to regulate and control the output bit rates of video sources in the network to achieve the best trade-off between quality and bandwidth utilisation (Girod, 1993). One of the main challenges of video communications is to provide a guaranteed quality of service when the network is swamped with excessive delays and informa- tion loss rates (Kurose, 1993). Network congestion could be avoided by using preventive instead of reactive remedies. Congestion avoidance techniques in video communications must consist of an efficient flow control mechanism that regu- lates the rates of active video sources (Jacobson, 1988). In a bit rate regulation scheme, the video source might sometimes be required to decrease its output flow due to high traffic load across the network. This reduction in bit rate could certainly lead to quality degradation since the quantisation distortion becomes more noticeable at lower bit rates. However, the quality degradation resulting from a coarser quantisation process is far less detrimental to the video quality than the effect of intolerable time delays and high data loss rates caused by a state of network congestion. Network congestion effects could also be more disastrous in real-time video services where the decoded video quality is much less tolerant to delay and data loss. Therefore, some policy must be adopted to prevent the Compressed Video Communications Abdul Sadka Copyright © 2002 John Wiley & Sons Ltd ISBNs:0-470-84312-8(Hardback);0-470-84671-2(Electronic) occurrence of congestion or reduce its effect in high traffic load conditions. A lot of research efforts have been exerted to establish efficient techniques for resolving congestion. Bolot and Turletti (1994) have developed a feedback control mechan- ism for flow control of video sources over the multicast backbone (Kumar, 1996) of the Internet. In this preventive rate control scheme, the rate control of a video encoder is regulated by modifying some encoding parameters, as indicated by some feedback messages sent by network receivers. Each receiver sends a feedback message that includes some statistics data such as average packet transit time, average loss rate for multicast traffic, average packet delay, etc. The sender collates this data and adjusts its output flow accordingly. Another feedback mechanism (Bolot, Turletti and Wakeman, 1994) employs a probing technique to solicit information and estimate the number of receivers in a multicast tree. A number of video scaleability paradigms (Radha et al., 1999; Stuhlmuller, Link and Girod, 1999; Horn and Girod, 1997) have been proposed for Internet streaming applica- tions. Other research efforts produced reactive approaches such as error conceal- ment and video data recovery schemes, which we will elaborate on in the next chapter. In this chapter, we present a variety of rate control algorithms that can be used in compressed video communications today. These algorithms can perform dynamically in accordance with the varying channel conditions. The status of the channel is reported back to the video source by a number of receivers that have special traffic data compilation capabilities. These feedback reports make the video source more network-aware and thus contribute to efficiently adapting the flow control algorithms to the reported channel conditions at any instant of time. 3.2 Bit Rate Variability of Video Coders All the standard video coding algorithms described in the previous chapter produce a variable bit rate per frame for a constant quantisation parameter. To guarantee a constant perceptual quality of the decoded sequence, it is necessary to keep a constant quantiser value Qp during the encoding process. Alternatively, varying the quantiser value on a frame or MB basis could achieve a constant output bit rate but at the expense of an undesirable variation in the decoded video quality. A new variable quantiser rate control algorithm has been proposed (Perra, Pinna and Giusto, 2000) to produce a minimal output bit rate for a fixed objective quality. The relationship between the temporal activity and quality of service in video communications is shown in Figure 3.1 for both fixed and variable bit rate encoding. In addition to the constant quality justification of variable rate video, the fluctuation of bit rates is also useful for the dynamic allocation of available bandwidth. As described in Chapter 2, a video source produces a higher output rate with a more active scene or more detailed texture. The drop in the output rate of a video source could be exploited to allocate a larger portion of bandwidth to a 76 FLOW CONTROL IN COMPRESSED VIDEO COMMUNICATIONS Temporal activity Time Quality Time Quality Time Bit rate Bit rate (b) Variable rate (fixed q uality) (a) Fixed rate (variable quality) Figure 3.1 Relationship between quality and bit rate more active source in the network, thereby ensuring a more efficient bandwidth sharing than for the fixed bandwidth scenario. However, this dynamic bandwidth allocation requires a flow control mechanism which can police and dictate the output traffic of each video source on the network in accordance with the time- varying network conditions and requirements. In general, there are two main reasons why a block-transform video coder has this variable bit rate characteristic. A digital video signal incorporates a huge amount of sequence-dependent redundancies in both time and space. The compression efficiency of a video encoder is determined by the amount of redundancy that is detected and sup- pressed from the video sequence in both the spatial and temporal domains. It is the proportional removal of these spatial and temporal redundancies which make the 3.2 BIT RATE VARIABILITY OF VIDEO CODERS 77 output bit rate a variable function of time. For instance, an MB in a predicted frame could represent an unchanged picture area between two successive frames. Therefore, this MB remains stationary as compared to the corresponding MB in the preceding frame. In this case, the block-transform video encoder does not code the MB for improved coding efficiency but sets a single bit flag (COD : 1) indicating to the decoder that this MB has been skipped in the encoding process. The number of uncoded MBs in predicted frames is certainly a function of the temporal correlations in the video content. This number also depends on the temporal similarities criteria used by the encoder as to whether a certain MB in a predicted frame is to be coded or skipped. The variability of the number of coded MBs in predicted frames certainly leads to a variable output bit rate. On the other hand, the spatial correlations between pixels of the same video frame dictate the number of bits required to encode the 64 transform coefficients of each 8 ; 8 block of data. This is in addition to the chosen quantisation parameter that controls the number of zero coefficients and non-zero levels that are fed into the run-length encoder. Obviously, since the quantised coefficients (TCOEFF) of the video blocks result in different levels and zero-run lengths, the run-length encoder produces a different number of VLC words (RUN, LEVEL) per block even when the quan- tisation parameter remains constant throughout the encoding process. Moreover, the temporal scaleability feature enabled by multi-layer coding, such as in MPEG- 4 for instance, contributes towards the variable output bit rate. Different VOP rates, frame skipping, different quantisation parameters per video layer, are all factors that contribute to this highly time-varying output bit rate. The second factor that leads to the bit rate variability in video coding algo- rithms is the presence of Huffman coding. Variable-length coding is used to optimise the compression efficiency by achieving an optimal average bit length per codeword. As opposed to fixed-length coding, Huffman coding attempts to assign a code to a certain event, such as a run of zeros, based on the likelihood of its occurrence. The more likely the event, the shorter the code and vice versa. For some video parameters defined by the syntax of a video coding algorithm, such as ITU-T H.263 (Refer to Appendix A), specific Huffman tables are defined. These tables are used to guarantee an optimal average number of bits per coded video parameter. However, due to spatial correlations of video data, different areas of a video frame could be coded at different compression ratios, hence with different number of bits, even if they happen to have an equal number of MBs and/or pixels. This could be best demonstrated by assigning variable-length codes to the differ- ent runs of zeros and non-zero levels produced by the run-length encoder. Table 3.1 lists the fixed and variable-length video parameters of the H.263 compression algorithm. Although the table shows more parameters that are fixed-length coded, the contribution of variable-length parameters to the overall output bit rate is much higher than that of fixed-length parameters. Therefore, the percentage of the bits corresponding to variable-length parameters is much higher than that of their fixed-length counterparts. This conclusion is better illustrated in Table 3.2 which 78 FLOW CONTROL IN COMPRESSED VIDEO COMMUNICATIONS Table 3.1 Fixed and variable length video parameters in H.263 coding algorithm Codes Layers Variable length Fixed length Picture Bit Suffing ESTUF, PSTUF Synchronisation PSC(22), ECS (22) Addressing TR (8), TRB (3) Quantisation step size PQUANT (5), DBQUANT (2) Administrative PTYPE (13), CPM (1), PSBI (2) Spare PEI (1), PSPARE (8) Group of Bit Suffing GSTUF Synchronisation GBSC (17) Blocks Addressing GN (5) Administrative GSBI (2), GFID (2) Quantisation step size GQUANT (5) Macroblock Administrative MCBPC, MODB, CBPY Administrative COD (1), CBPB (6) Motion MVD, MVD2-4, MVDB Quantisation step size DQUANT (2) Block DCT Coefficients (except Intra DC terms) TCOEFF DC terms of Intra DCT Coefficients INTRADC (8) shows that most of the bits of an H.263 stream, for the Foreman sequence coded at 30 kbit/s, are due to the variable-length codes. More precisely, the statistics show that the DCT coefficients (excluding the fixed-length INTRADC codes) and the differential MV components contribute to 75 per cent of the overall output flow of the encoder. 3.3 Fixed Rate Coding Although a variable bit rate is sometimes desirable for dynamic bandwidth allocation, constant bit rate transmissions are useful for fixed bandwidth channels such as PSTN. To achieve fixed rate video transmissions, a buffer between the video encoder and the channel is used to smooth out the bit rate fluctuations. Obviously, buffering the compressed video streams before transmission entails a certain amount of delay, which must be avoided or at least minimised in real-time video services. This buffer could only regulate the output bit rate for short-term variations. In some video sequences, bit rate fluctuations could last for several frames and thus a large buffer would then be required to absorb long-term 3.3 FIXED RATE CODING 79 Table 3.2 Contribution of video parameters to overall bit rate for Foreman coded by H.263 at 30 kbit/s Fixed Variable Total Synchronisation PSC 0.73 5.25 GBSC 4.53 Addressing TR 0.27 1.60 GN 1.33 Quantisation PQUANT 0.17 1.50 GQUANT 1.33 Administrative PTYPE 0.43 16.67 CPM 0.03 GFID 0.53 COD 3.27 CBPY 8.86 MCBPC 3.55 DCT coefficients INTRADC 5.07 46.42 TCOEF 41.35 Motion vectors MVD 28.52 28.52 Spare PEI 0.03 0.03 Total 17.72 82.28 100.00 fluctuations. This long-term buffering introduces intolerable details and makes the provision of real-time video services impossible. Therefore, in addition to buffering the video data, other measures need to be taken in order to reduce the burstiness of the output flow of video coders. The most commonly used technique is to adjust some video encoding par- ameters as a function of the buffer fullness, i.e. by feedback control. On the other hand, the use of current picture activity, i.e. feed-forward control, provides an alternative means of indicating to the video coder the need to adjust the encoding parameters. The buffer-based approach for bit rate regulation is depicted in Figure 3.2. In the next section, we describe the response of the video coder to feedback or picture activity feed-forward messages. 3.4 Adjusting Encoding Parameters for Rate Control Any attempt to control the output bit rate of a video coder involves trading-off quality and compression efficiency. Reducing the bit rate could be done at the expense of degraded quality. In block-transform video coders, there are four different encoding parameters which could be adjusted to control the output bit 80 FLOW CONTROL IN COMPRESSED VIDEO COMMUNICATIONS Modify Coder Parameters Source Coder Buffer Buffer status Picture activity measure channel To Input Figure 3.2 Feedback and feed-forward approaches in buffer-based video bit rate control systems rate. Firstly, the frame rate, which determines the number of encoded frames per second, is one encoding parameter that could be modified to match the bit rate requirements. Since the frame rate control method targets the temporal and not the spatial redundancies of video signals, it is generally used when the quality of individual pictures cannot be compromised. Another possible way to modify the output bit rate is to encode only a spatial portion of each 8 ; 8 block of pixels such as the diagonal coefficients (1 ; 1), (2 ; 2), etc., or only the low-frequency coeffi- cients of a block. Fewer bits are then produced per block at the expense of reduced quality due to the removal of more video data. To optimise quality and preserve the block perceptual fidelity, the DC coefficient, which contains the largest portion of the block energy, has to be coded and AC coefficients could be dispensed for lower output rates. If the motion aspect of a video scene has an important contribution to the overall quality of video then the spatial video quality could be compromised for a better temporal video quality. In this case, the frame rate is preserved for a coarser quantisation of spatial video details. The third parameter, which can be adjusted for controlling the video bit rate, is the quantisation parameter Qp. This par- ameter controls the number of bits required to quantise output video codewords, such as transform coefficients. Increasing Qp results in encoding the DCT coeffi- cients with fewer bits, since more zero coefficients would then be obtained (due to quantisation) prior to run-length coding. However, lower Qp values lead to a wider encoding range and hence higher bit rates. Adjusting the quantisation step-size could be done on a frame, GOB or MB basis. Figure 3.3 shows that the number of bits per frame of an H.261 coded sequence at a resolution of 352 ; 240 varies inversely with Qp values. The fourth encoding parameter that can be manipulated to control the output bit rate of a video encoder is the motion detection threshold. This threshold is set to control the decision of whether an MB in a predicted frame (P-frame) is coded (COD : 0) or skipped (COD : 1). If the threshold increases, the encoder becomes less sensitive to motion and thus the number of coded MBs decreases. Therefore, the number of bits required for encoding a P-frame decreases at the expense of 3.4 ADJUSTING ENCODING PARAMETERS FOR RATE CONTROL 81 Figure 3.3 Number of bits per frame for a video sequence of 150 frames, with a resolution of 352 ; 240, coded with H.261 at different Qp values and a fixed motion threshold of 2.2 lower sensitivity to motion. Conversely, for a lower motion threshold a larger number of MBs will be coded, leading to an improved motion sensitivity but higher bit rates. Similarly, the INTRA/INTER mode decision threshold could also be used to control the output bit rate of each coded MB in a predicted frame. More INTRA coded MBs lead to increased bit rates but improved decoded quality. The improved quality of INTRA coded MBs is mainly due to the absence of prediction in this coding mode. Figure 3.4 shows the output number of bits per frame for the same video sequence as in Figure 3.3, encoded with H.261 at different motion threshold values. The aforementioned four encoding parameters could be adjusted during the encoding process to control the output bit rate of a video encoder. The adjustment of the parameters is usually done in line with the channel status that is periodically reported to the video source. The regulation of the encoding parameters leads to a variable level of perceptual quality, but this could only have a graceful effect as compared to quality degradation resulting from congestion. Most video com- munication systems that rely on adjusting the video encoding parameters as part of controlling the output rate adopt preventive flow control techniques (Dagiuklas and Ghanbari, 1992). In these techniques, the rate control system remains active to prevent the network from reaching a state of congestion, hence the name preven- tive. 82 FLOW CONTROL IN COMPRESSED VIDEO COMMUNICATIONS Figure 3.4 Number of bits per frame for a video sequence of 150 frames, coded with H.261 with Qp : 10, for different motion threshold values 3.5 Variable Quantisation Step Size Rate Control The traditional approach to regulate the output bit rate of a video source is to adjust the quantisation step size of the next frame, GOB or MB, based on the local buffer occupancy that is essentially dictated by the status of the network. However, although varying the quantisation step size affects the output rates, the average number of bits generated for each frame (GOB or MB) is not linearly dependent on the quantisation step size, as shown in Figure 3.5. For instance, when Qp is less than 5, a unity variation can produce two to five times more output video data. Conversely, the same unity change in Qp may generate only a few dozen more bits when the quantisation parameter exceeds 20. In addition to that, the video content affects the number of bits required to code a video frame. Therefore, classical quantisation rate control techniques provide unpredictable and sometimes highly fluctuating bit rates, thereby increasing the likelihood of local buffer overflow that results in severe data losses in the case of network congestion. In order to produce a stable video output, more sophisticated rate control algorithms have to be employed. In these algorithms, both the buffer fullness and the picture activity have to be used to choose an appropriate quantiser parameter Qp so that the resulting bit rate is close to the target bit rate. 3.5 VARIABLE QUANTISATION STEP SIZE RATE CONTROL 83 0 102030 Quantiser step size 0 5000 10000 15000 Data rate per frame (bits) Figure 3.5 Average data rate per frame as a function of quantiser 3.5.1 Buffer-based rate control One widely accepted buffer-based rate control technique is called the scaleable rate control (SRC) algorithm (ISO/IEC 14496, Annex L) for real-time MPEG-4 video transmissions. SRC is designed to achieve scaleability at various bit rates from 10 kbit/s up to 1 Mbit/s, and various spatial and temporal resolutions. This technique can handle I, P and B frames and can only be applied for single visual object (VO) rate control purposes. The SRC scheme assumes that the encoder rate distortion function can be modelled as: R : X  ; S ; Q\ ; X  ; S ; Q\ where R is the encoding bit count, S is the encoding complexity (mean absolute difference), Q is the quantisation parameter, and X  and X  are the modelling parameters. The SRC scheme procedure divides its main processes into four stages: in- itialisation, computation of the target bit rate before encoding, computation of the quantisation parameter Qp before encoding, and updating the model parameters based on the results obtained from coding the current frame. Firstly, the SRC algorithm checks whether the current frame is an INTRA or INTER frame. For INTRA coded frames, the initialisation part extracts the first and second order coefficients and Qp is set to the value initially specified (by the user or the application). SRC then skips steps 2 and 3 and the rate distortion model par- 84 FLOW CONTROL IN COMPRESSED VIDEO COMMUNICATIONS [...]... COMPRESSED VIDEO COMMUNICATIONS resulting video layers consist of one base layer (BL) and a number of enhancement layers (EL), all of which have different contributions to decoded video quality The base layer is essential for the reconstruction of the video sequence, while enhancement layers help improve the perceptual quality but their absence causes a graceful deterioration of the received video quality... temporal video quality On the other hand, transform coefficients occupy a larger portion of the coded bit stream, as indicated in Section 3.2; therefore, dropping these coefficients in the case of congestion helps reduce the flow rate of the video encoder with only a graceful degradation of the reconstructed video quality Consequently, the output video parameters (e.g MVs and DCT coefficients) of a standard video. .. 33.12 34.56 28.01 37.41 14.50 38.16 37.89 14.35 These video codewords are ordered according to the syntax of the video coding algorithm and then sent to a local buffer before transmission In the case of network congestion, these buffered codewords could be subject to excessive time delays Since real-time video is highly delay-sensitive, the delayed video codewords are arbitrarily dropped off the local... the delayed video data is dropped at the encoder side, the synchronisation between the video encoder and decoder is maintained by sending a non-coded flag (COD : 1) for each dropped MB This means that the delayed MB data is then replaced by a single bit which requests the decoder to skip the corresponding MB during the video reconstruction process However, the 96 FLOW CONTROL IN COMPRESSED VIDEO COMMUNICATIONS... SCR algorithm passes the results to the model updating stage in order to compute the new model parameters, and the procedure continues In addition to SRC, another quantisation control scheme adopted in the MPEG-2 video coder is the Test Model 5 (TM5) algorithm [TMOD] for rate control purposes TM5 describes a procedure for controlling the bit rate by adapting the quantisation parameter of an MB, and it... congestion have different sensitivities to information loss This difference could be better observed by evaluating the effect of information drop on one type of video parameters at a time This is similar to partitioning the coded video data into two parallel sub-streams, each containing one type of video parameters, and applying the information drop on one sub-stream at a time Figures 3.16 and 3.17 depict... their contribution to overall video quality Assigning these priorities must be done based on a complete study of the sensitivity of individual video parameters to information drop (Leicher, 1994) This technique has to account for all the parameters of a video frame including the administrative data used to enable the synchronisation of the decoder The prioritisation of video parameters could then become... the sensitivity of video parameters and the reported channel conditions at one instant of time When the network is reported to be in good condition, the video coder guarantees the transmission of all the coded information by setting all priority 3.7 RATE CONTROL USING PRIORITISED INFORMATION DROP 99 Figure 3.18 Time-varying factors that affect the dynamic prioritisation algorithm of video parameters levels... improvement brought by this technique becomes marginal If the video encoder has a preventive approach to network congestion, ILB has the role of estimating the required number of bits to be discarded as a trade-off between video quality and rate/congestion control To illustrate the efficiency of the local feedback loop technique in controlling the video rate without compromising quality, we will consider... section 3.6 Improved Quality Rate Control Using ROI Coding In some kinds of video sequences, a priori knowledge about the content of the video scene could be exploited for improved coding efficiency by coding the regions of interest (ROI) more accurately than the rest of the video content For instance, in head-and-shoulder types of video sequence, one tends to concentrate on the face, giving more emphasis . a video scene has an important contribution to the overall quality of video then the spatial video quality could be compromised for a better temporal video. compared to quality degradation resulting from congestion. Most video com- munication systems that rely on adjusting the video encoding parameters as part