Yes, every emperor had Augustus as part of his name. They all also used the name Caesar. The first three had it by right (Augustus by adoption by Julius Caesar, Tiberius by being adopted by Augustus, and Caligula because his father Germanicus was adopted by Tiberius). Claudius was not a Caesar but when he became emperor (on the assassination of Caligula) he added Caesar to his name.
Nero was born Lucius Domitius Ahenobarbus, but was adopted by Claudius. Through his mother he was a direct descendant of Augustus.