The only traces I can find today of the "Jubilee of Pole Bearers" are references to Forrest's speech. It may have been more of a social or fraternal organization, rather than a political group.
Forrest died of complications of diabetes in 1877. Maybe he was already ailing when he addressed the group. He'd gone bankrupt some years earlier when his railroad failed.
So perhaps Nathan Bedford Forrest was trying to get right with God. He may have felt guilty about something: "Men have come to me to ask for quarter, both black and white, and I have shielded them."
That adds something to our understanding of the man. But if Forrest helped create the conditions where White Southern politicians couldn't talk like this publicly to a Black organization for a century, that also shouldn't be ignored.
Just because you can't find anything else doesn't mean they didn't exist. Forrest wasn't even the keynote speaker. The event was barbecue held at a Memphis county fair, by the Independent Order of Pole-Bearers.