The Congressional Research service has a handy paper on that:
Recall of Legislators and the Removal of Members of Congress from Office
SummaryUnder the United States Constitution and congressional practice, Members of Congress may have their services ended prior to the normal expiration of their constitutionally established terms of office by their resignation or death, or by action of the house of Congress in which they are a Member by way of an “expulsion,” or by a finding that in accepting a subsequent “incompatible” public office, the Member would be deemed to have vacated his congressional seat.
Under Article I, Section 5, clause 2, of the Constitution, a Member of Congress may be removed from office before the normal expiration of his or her constitutional term by an “expulsion” from the Senate (if a Senator) or from the House of Representatives (if a Representative) upon a formal vote on a resolution agreed to by two-thirds of the Members of that body present and voting. While there are no specific grounds for an expulsion expressed in the Constitution, expulsion actions in both the House and the Senate have generally concerned cases of perceived disloyalty to the United States, or the conviction of a criminal statutory offense which involved abuse of one’s official position. Each house has broad authority as to the grounds, nature, timing, and procedure for an expulsion of a Member. However, policy considerations, as opposed to questions of authority, have appeared to restrain the Senate and House in the exercise of expulsion when it might be considered as infringing on the electoral process, such as when the electorate knew of the past misconduct under consideration and still elected or re-elected the Member.
As to removal by recall, the United States Constitution does not provide for nor authorize the recall of United States officers such as Senators, Representatives, or the President or Vice President, and thus no Member of Congress has ever been recalled in the history of the United States.
It would most likely be done at the state level - a governor would remove a serving Congress person or there would be a recall election. I don't know about the history of such. Of course the Congress can refuse to seat an individual - as the Dems are doing now in the House.