Mr Goje withdrew from the race after meeting with President Muhammadu Buhari on Thursday at the presidential villa, Abuja.
Several efforts to oust him from his position often met brick walls with the Senate President atimes outsmarting his opponents.
The decision of the opposition members may further boast the chances of Femi Gbajabiamila to succeed Yakubu Dogara, the outgoing speaker who is no longer interested in the position.
Mr Dogara stated this in reaction to online reports wherein he is said to be interested in retaining his seat at the lower chamber of the National Assembly.
Mr Buhari had on Monday while signing the 2019 budget blamed the National Assembly for delaying in passing the budgets sent to it.
The opposition Peoples Democratic Party (PDP) is also warning up to thwart the wish of the APC does not manifest at the both chambers of the National Assembly.
The top lawmaker also said despite his relationship with president Muhammadu Buhari, he will not be a rubber stamp if the Nigerian leader.
Emefiele appointed CBN Governor on June 2, 2014, is due to end his first term on June 2, this year.
With his interest in the race, the Borno lawmaker is challenging another lawmaker the party adopted as its preferred candidate for the position.
The N8.83 trillion budget was presented to a joint session of the National Assembly by Mr Buhari on December 19, 2018.