How can I remove a speaker from a session in the agenda?

You can remove Speakers from sessions, under Event Content > Agenda > Session Manager, by clicking on the pencil icon of a session and clicking the trash icon next to the specific Speaker.

Under Event Content > Speaker Center, you can also remove sessions by clicking on the Speaker, and clicking the trash icon of that specific session.