“Also a lot of business applications have to play well with legacy systems, that have a bunch of quirks, that cannot simply be deduced by using AI.”
It can when trained to include those “quirks”.
I have some significant experience working with our in-house, sandboxed AI. For example, I have had our AI produce a series of Automated Unit Tests (to be differentiated from standard 'unit tests'). Creating of AUTs require that they follow the template Arrange-Act-Assert.
What I got back looked pretty good, and at first glance, seemed like it may have saved our development team hundreds of hours of development and testing....
....except....
... it only provided the bare minimum of AUT code. The stuff was not sufficient to test our business use-cases. Sure, it provided my asked-for 80% code coverage... but poorly.
The amount of prompt-engineering it would take to produce good AUTs would require a book about the thickness of 'War and Peace'. And you cannot verbally prompt-engineer, since invariably you will either leave something out or you will need to revise the prompt. This means it really has to be written.
So, to your point, training it to catch all the quirks is an extremely daunting and time-intensive task, and that may prove impractical. Furthermore, that's even if you are AWARE of every quirk. Your development team may not.
I know you pretty well, you like to be right. I ask, however, that you accord me subject-matter-expert deference in this particular topic, as I would accord you the same deference in, say, operation of a nuclear plant... which I believe you have SME experience in.