I mean, a mile is very small compared to Earth. This is akin to a penny shredding your house if it falls from the sky.
Because it is going really fast.
Remember, an object coming in from space is going to be traveling at extremely high velocity. When the object suddenly decelerates upon impact or when the sheering forces tear it apart, its massive amount of kinetic energy (energy of motion) is instantly converted into thermal energy (heat energy). This instant conversion creates an explosion that can be as great or even thousands of times greater than the largest nuclear weapons ever built.
The event at Tunguska was estimated to have been caused by an object roughly 50-100 meters in diameter. Far less than 1+ miles in side. While Tunguska thankfully didn't hit the ground, it detonated in the air around 15,000-25,000 feet. When it exploded, it was traveling around 7 miles per second. The force of the explosion has been estimated to be anywhere from 15 megatons up to 40 megatons. (Most experts agree somewhere around 20-25 megatons is the most likely.) Think about how much damage something like this could do if it hit the ground or a populated area. Hiroshima was only 12.5-15 kilotons. (1,000 kilotons=1 megaton) The explosion at Tunguska also created an detectable electromagnetic pulse which damaged electric equipment (telegraphs, phone systems, wireless) at very long distances from the blast site.
We all know about Meteor Crater in Arizona. A somewhat smaller object created this giant hole.
Now compare it to Sedan Crater which is at the Nevada Test Site. It was created by a 104 kiloton test (around 1/10 of a megaton) in 1962.
Contrary to popular theories, size isn't everything.
At Hiroshima, just a gram of uranium was converted into energy. (Yeah, different processes, but speed can be a similar multiplier. A quarter travelling at the speed of light could do a lot of damage were it to impact the earth.