The rotary inverted pendulum (RIP) has a fourth order nonlinear dynamics, and its control is typically with two stages, i.e., a swing-up control and a balancing control. For Linear-quadratic regulator (LQR) control, under large disturbances, the RIP stability may be lost. With Input-output feedback linearization (IOFL) control, the linearized portion’s stability is governed by the linear control scheme, and the closed loop for the unlinearizable portion of the original dynamics forms the so-called internal dynamics, which not necessarily has the bounded input bounded output (BIBO) stability. A modified IOFL control is presented with these approaches: for the linearized portion, the gains associated with the angle and angular velocity of the inverted pendulum are selected based on enhancing the stability via pole placement; for the unlinearizable portion, the gains associated with the angle and angular velocity of the rotary arm are determined by minimizing the cost function via genetic algorithm. Therefore, the pole-placement genetic algorithm synthesized IOFL control promises an augmented stability region for the RIP system. The results can be generalized in the applications of various nonlinear systems’ control design.