CHIEF government whip Joel Fitzgibbon says Labor powerbrokers are not counting numbers despite heightening leadership speculation in Canberra.
But in a frank interview with Fairfax on Wednesday, Mr Fitzgibbon conceded nervous caucus members were talking about the leadership and Labor's electoral fortunes.
"It would be silly to tell people watching your program there's nothing going on," Mr Fitzgibbon told Breaking Politics host Tim Lester.
"Obviously, internally, people are looking at the polls and they're expressing concerns about the future of the government and, indeed, the party and you'll get conversations and those conversations are, unfortunately, making their way into the media.
"We should keep them internal."
Mr Fitzgibbon said Labor's poor showing in just about every opinion poll was causing "unrest" in the caucus, particularly among MPs in marginal seats.
But he said it was nothing more than speculation at this stage, and had seen nothing that would suggest there was going to be a change of leadership this week.
Meanwhile, Prime Minister Julia Gillard and Kevin Rudd were noticeable absentees at a breakfast function in Parliament House this morning.
Indigenous Employment Minister Julie Collins apologised for Ms Gillard's no-show at the Generation One event, saying she had been "delayed".
A spokesman for Mr Rudd told APN Newsdesk he had decided on Tuesday not to attend.
"Our office advised the organisers yesterday that Mr Rudd would no longer be attending as he wanted to ensure the focus was, appropriately, on the great work that Generation One are doing," the spokesman said.
"Mr Rudd is meeting with Mr Forrest to discuss Generation One today."
Ms Gillard's office told News Limited she had planned to "pop in" but it had become "too busy".
Meanwhile, key independent Tony Windsor said Labor risked losing his support if dumped Ms Gillard as leader.
Mr Windsor and fellow independent Rob Oakeshott helped Ms Gillard form government in the wake of the 2010 election by guaranteeing confidence and supply.
"The document is not transferable," Mr Windsor told Sky News.
He said the PM had succeeded in maintaining the Parliament and providing stable government.
"If the Labor Party don't think she's done that and want to make their changes they're quite entitled to do that. But they can't expect me to be some sort of fellow traveller while they work out their stability and instability issues."
He predicted Mr Rudd, who was unceremoniously dumped as Labor leader in June 2010, had "Buckley's" of retaking the leadership.
Update your news preferences and get the latest news delivered to your inbox.