My understanding of this has been wrong until recently. But as I like the P-38 I have been doing research and now have a good grasp of the phenomena. And while this is about the P-38, it applies to all aircraft that approach mach 1. Just that the Lightning was the first aircraft to do so, so it hit the snags first.
The misconception that I have been operating under is that tail redesign and hydraulics were part of the solution to compressibility, that is not the case. The Lighting had two problems with high speed dives. The first one solved was tail flutter, in which when the P-38 entered a high speed dive, the tail would vibrate, and sometimes break apart causing a crash. This problem was also encountered on the P-47, P-51, and Spitfire, and probably other planes as well. This was a simple aerodynamics problem and readily fixed.
The second and tougher problem was compressibility, where the controls would lock in a dive, tail would vibrate and fall off sometimes, and the plane would go into an inverted loop, usually breaking up from the stress. This came from doing a steep high speed dive from about 35,000 feet, and if a pilot was lucky the plane would hold together until he could gain control back at 10,000 feet.
When the Lightning dived from 36,000, it could hit speeds upwards of 0.7 mach. That is the speed of the plan itself. The air going across the top of the wing goes faster, which causes lower pressure resulting in lift. At around 0.7 mach, the air going across the top of the wing hits mach 1. You don't get a big sonic boom, but you get many tiny sonic booms, one of the effects of which is it dumps the air off the wing causing a total loss of lift.
The tail now becomes the only part of the plane with lift, which tends to push the plane into a steeper dive. If the pilot pulls back on the stick to pull out of the dive, the lift of the tail is increased, further increasing the dive angle. One pilot survived without his plane breaking up and completed an inverted loop.
The second effect of the tiny sonic booms is continuous pressure waves, which causes metal fatigue and structural failure in the tail. This is the reason that the engineers thought the tail flutter and compressibility were the same problem for some time, and it still gets reported as such. The symptoms are quite similar.
The planes would pull out at 10,000 feet because of simple physics. At 36,000 feet mach 1 is 540 mph, at 10,000 feet it is 620 mph. Even though the plane is going the same speed or faster, the air over the top wing is no longer hitting mach 1 and there are no more sonic booms, and flight returns to normal.
The fix to it was easy, and it is still employed today. A dive flap is added behind the leading edge of the wings outside the engine nacelles. It is a small piece of metal that comes out at a 15 degree angle to the wing, increasing lift. As long as the wings maintain lift, no problems.