Po-Chih ChenBorching SuYenming HuangBORCHING SU2019-10-312019-10-3120171053587xhttps://scholars.lib.ntu.edu.tw/handle/123456789/430195https://www.scopus.com/inward/record.uri?eid=2-s2.0-85023781968&doi=10.1109%2fTSP.2017.2718971&partnerID=40&md5=4a03d07608a7b70f298373e828f4d39fIn this paper, a new matrix-based characterization of generalized-frequency-division-multiplexing (GFDM) transmitter matrices is proposed, as opposed to traditional vector-based characterization with prototype filters. The characterization facilitates deriving properties of GFDM (transmitter) matrices, including conditions for GFDM matrices being nonsingular and unitary, respectively. Using the new characterization, the necessary and sufficient conditions for the existence of a form of low-complexity implementation for a minimum mean square error (MMSE) receiver are derived. Such an implementation exists under multipath channels if the GFDM transmitter matrix is selected to be unitary. For cases where this implementation does not exist, a low-complexity suboptimal MMSE receiver is proposed, with its performance approximating that of an MMSE receiver. The new characterization also enables derivations of optimal prototype filters in terms of minimizing receiver mean square error (MSE). They are found to correspond to the use of unitary GFDM matrices under many scenarios. The use of such optimal filters in GFDM systems does not cause the problem of noise enhancement, thereby demonstrating the same MSE performance as orthogonal frequency division multiplexing. Moreover, we find that GFDM matrices with a size of power of two are verified to exist in the class of unitary GFDM matrices. Finally, while the out-of-band (OOB) radiation performance of systems using a unitary GFDM matrix is not optimal in general, it is shown that the OOB radiation can be satisfactorily low if parameters in the new characterization are carefully chosen. © 1991-2012 IEEE.characteristic matrix; Generalized frequency division multiplexing (GFDM); MMSE receiver; orthogonal frequency division multiplexing (OFDM); out-of-band (OOB) radiation; prototype filters; unitary matrixBandpass filters; Characterization; Frequency division multiplexing; Mean square error; Multipath propagation; Radio receivers; Receivers (containers); Transceivers; Transmitters; Characteristic matrices; Complexity theory; Generalized frequencies; MMSE receivers; Out-of-band; Prototype filters; Prototypes; Unitary matrix; Orthogonal frequency division multiplexingjournal article10.1109/tsp.2017.27189712-s2.0-85023781968