[AlexB17]

Ну переводить не буду уж - но в кратце вот:

Trellis quantization optimizes the motion estimation following a rate distortion function. It slightly decreases encoding speed and leads to a slight quality loss along (but the decrease in used bitrate is considerable in comparison).

Trellis quantization is actually very dependent on bitstream format: It's purpose is to reduce some inefficiencies in Huffman coding of DCT coefficients. But H.264 uses Arithmetic coding, which doesn't suffer the same problems.

Trellis uses rate/distortion optimization, which means that it may decrease the quality of the video if it can save enough bits to be "worth it" (where the definition of worth depends on quantizer). Or it might have increased the quality and you could be underestimating the improvement by looking at only filesize. The tradeoff threshold is supposed to be calibrated such that it doesn't modify quality much, but it's not perfect, and can only be accurately measured in 2pass mode. (The same advice applies to all options, but especially vhq, bvhq, and anything related to B-frames.)