I definitely think rallying is the toughest racing type.
I can't say from experience other than ten years of playing rally racing games... I remember distinctly at five years old, playing Gran Turismo 2 on the PS1 with my dad, rally racing with the wheel and pedals....
I remember dad drifting his Audi Coupe GT in the snow, nearly sideways through corners, sitting in the back seat....
Thing about rallying is quick reflexes, low light situations, corners in very fast relation to each other, bumpy or unstable (sand, gravel, etc.) surfaces, water.... Things you don't get in NASCAR, where all the cars are more or less the same, driving on a flat, well lit track.