In some ways, the Pentium 4 represents what many may view as "backward steps" in technology - some have viewed the increasing instruction latencies as a negative side-effect of the quest for more MHz. This has been happening since the days of the 486, perhaps earlier, due to pipelining.
Intel has also done away with the traditional instruction cache in favor of the new Trace Cache. They have chosen to use fewer functional units than the competition, and knowingly reduce average IPC (Instructions Per Clock). In the quest for performance, the best metric is time of completion for a task. It matters not, in the end, whether a processor achieves greater performance through higher IPC (instructions per clock), or greater frequency. In the end, absolute performance is what matters.
Intel has chosen to sacrifice a little bit of average IPC, in exchange for radically higher clock speeds than their previous architecture when on the same process technology. While the Pentium 4 at 1.3ghz doesn't outperform the 1ghz Pentium III in many tasks, the highest performing Pentium 4, at 1.8ghz, is indisputably faster than the fastest Pentium 3 (at 1.1ghz - which was a silent introduction, I might add). The Pentium 4, when in the same process technology as the Pentium III, will have greater absolute performance in nearly every facet of computing (performance/watt excluded).
Of course, the Pentium 4 has more to ward off in performance than it's older sibling, the Pentium III, it also has its archrival AMD's Athlon core to contend with. While the Pentium 4 may not radically outperform the Athlon series right now (and in some cases, at all), it has a future, which indicates it has that potential. With Intel being at least a year ahead of AMD in terms of volume .13-micron process technology, and Intel will be able to introduce much more highly clocked Pentium 4's at a pace much faster than AMD will be able to do, due to the difference in process technology.
AMD's current hope for survival rests with their Palomino core, which boasts even greater average IPC than the Thunderbird (which itself is higher than that of the Pentium 4). The Pentium 4 contains a new core intended to supplant the old, and venerable Pentium Pro core in all markets (over its lifetime, not right away). As such, it was designed with many aspects of the future in mind, such as the soaring disparity between main memory and CPU clock-rates. Its performance now is relatively unimportant (as long as it outperforms the Pentium III in the same process technology), which wasn't true of the Athlon (AMD needed the Athlon to crush the Pentium III right then and there to survive as a company). There are technical indications that the Athlon core isn't as future-proof as the Pentium 4, which means that the battle between the two major x86 rivals will spark even hotter.
To see how the Pentium 4 really performs, check out latest review of the 1.7GHz flavor.