This paper is aimed to propose a controller which can achieve the tracking control of the rotary arm and the stability control of the inverted pendulum in the Rotary Inverted Pendulum (RIP) system. An Input-Output Feedback Linearized (IOFL) RIP system dynamics can be divided into two separate portions: one is the linearized dynamic portion associated with the pendulum arm, and the other is the un-linearizable portion associated with the rotary arm. For the unlinearizable portion, a fractional order PIλDμ (FOPID) controller is implemented to enhance the tracking control of the rotary arm with controller parameters determined from solving a prescribed optimization problem using Particle Swarm Optimization (PSO) algorithm. For the linearized portion, the state feedback control gains are obtained by conventional pole-placement method. It is revealed that a much more robust and effective tracking and stabilizing performance can be guaranteed with the FOPID involved in the controller design when compared with a previously designed synthesized IOFL controller for the RIP system. The new FOPID controller developed in this paper for the RIP system can be used as a reference to the tracking and stabilizing control of other complicated dynamic systems since the RIP system is a testbed for controller design of the fourth order Single-Input-Multiple-Output (SIMO) dynamic system.