Um....if your Neighbor’s fence is in YOUR yard, I’d say, it’s time he MOVED his fence back onto HIS property!
Get the property surveyed. If his fence is in your yard then you have a right to remove the fence or plant what you want by it or give him the option to move his fence to his property.