Day-to-day, the speed difference between the iPhone 4 and the iPhone 3GS isn't that different. But in more intensive apps and games? The iPhone 4 smokes the 3GS and is nearly as fast as the iPad.
When we used basic apps - Notes, Calendar, Weather, etc - performance between the iPhone 4 and 3GS was nearly the same. The iPhone 4 feels smoother - sliding from home page to home page is more fluid - but it's not dramatically different.
But fire up intensive apps and games like Plants vs Zombies and you see a real performance boost. The iPhone 4 flies past the 3GS. The iPhone 4 (on the right) had finished loading Plants vs Zombies before the iPhone 3GS (on the left) even got rolling. With other apps like Epicurious, it's the same story. It's not even close.
The better comparison with the iPhone 4 is the iPad. Given that they use the same A4 chip, it's not too big a surprise that they're fairly similar in performance. Look at how fast and accurate both the iPhone 4 and iPad determine their location. Yes, the iPad still has a little more instantaneous feel to it - hitting the home button exits apps quicker, sliding from page to page is smoother - but in pure app performance, the iPhone 4 is right there with the iPad, typically only trailing one or two seconds behind.
Considering the iPhone 3GS was one of the fastest phones on the market, the speed gained in the iPhone 4 is quite remarkable. And compared to the heavyweight iPad, the lightweight iPhone 4 actually holds its own. Having the iPhone 4 on iOS 4.0 versus the iPad's iOS 3.2 is probably a big reason for the boost, so we might see a different situation when the iPad upgrades to 4.x but for now, when iPad owners move "down" from a size perspective to use an iPhone 4, they're pretty much moving laterally from a performance aspect.