Shape optimization on mufflers within a limited space volume is essential for industry, where the equipment layout is occasionally tight and the available space for a muffler is limited for maintenance and operation purposes. To proficiently enhance the acoustical performance within a constrained space, the selection of an appropriate acoustical mechanism and optimizer becomes crucial. A multi-chamber side muffler hybridized with reverse-flow ducts which can visibly increase the acoustical performance is rarely addressed; therefore, the main purpose of this paper is to numerically analyze and maximize the acoustical performance of this muffler within a limited space. In this paper, the four-pole system matrix for evaluating the acoustic performance - sound transmission loss (STL) - is derived by using a decoupled numerical method. Moreover, a simulated annealing (SA) algorithm, a robust scheme in searching for the global optimum by imitating the softening process of metal, has been used during the optimization process. Before dealing with a broadband noise, the STL's maximization with respect to a one-tone noise is introduced for the reliability check on the SA method. Moreover, the accuracy check of the mathematical models with respect to various acoustical elements is performed. The optimal result in eliminating broadband noise reveals that the multi-chamber muffler with reverse-flow perforated ducts is excellent for noise reduction. Consequently, the approach used for the optimal design of the noise elimination proposed in this study is easy and effective.