Actually, the verse is "no man is able to pluck them out of my Father's hand." And no one can pluck us out of His hand. We can, however, choose to reject God.
Your interpretation goes to one extreme and, again, leaves us with no responsibility at all. However we get to this point, we can know that somewhere we made a wrong turn.
No one. Not even oneself.
One's faith is fickle. You can choose right one day, but the next you will not. There's no way this kind of faith is able to save. It's only faith given by God which can save.
Yes, you do have a responsibility to honor God by walking with Him. If you do not, He will discipline you. You can be a bad child; but you are a child nonetheless. Like the earthly family, once you are a part of a family you are included whether you like it or not.