Yeah, but this obviously careless remark/position has been overstated. Luther was exasperated at the fact that James's practical illustration of saving faith had gotten elevated to the level of high doctrine (e.g., Paul's systematic discussion of justification in Romans).
On the other hand, some of Luther's finest sermons are from the Epistle of James. (The RCC never tells its members that.)