Double-talk (DT)-robust acoustic echo cancellation (AEC) and acoustic feedback cancellation (AFC) are needed in speech communication systems, e.g., in hands-free communication systems and hearing aids. In this paper, we derive a practical and computationally efficient algorithm based on the frequency-domain adaptive filter prediction error method using row operations (FDAF-PEM-AFROW) for DT-robust AEC and AFC. The proposed algorithm features two main modifications: (a) the Wiener variable step size (WVSS) and (b) the gradient spectral variance smoothing (GSVS). In AEC simulations, the WVSS-GSVS-FDAF-PEM-AFROW algorithm obtains outstanding robustness and smooth adaptation in highly adverse scenarios such as in bursting DT at high levels, and in a change of acoustic path during continuous DT. Similarly, in AFC simulations, the algorithm outperforms state-of-the-art algorithms when using a low-order near-end speech model and in colored non-stationary noise.