The late, great Hunter S. Thompson wrote an article in 1970 entitled "The Kentucky Derby is Decadent and Depraved." Not surprisingly, he was right.
The infield at the Kentucky Derby has very little to do with horse racing. Instead, it's more about drinking, gambling, and people shedding clothes at a rapid rate.
While the track sells beer and other drinks, the libation of choice is of course the mint julep. Juleps are sold in commemorative glasses which you are allowed to keep, which is good for remembering later how many you actually had (for me, I think it was 5 or 6).
Most of the infield people were dressed normally, in contrast to those in suits and dresses. Some of the nicely clothed people were in the infield though, and they were rewarded with muddy clothes, as Friday turned certain parts of the ground into a quagmire. This was surprisingly amusing to watch.
At some point, you go to the betting windows to make your Derby wagers. Preferably, you want to do this as early as possible once you get in so it makes sense. For me though came the biggest problem of the day. Having already waited in line for a half hour, I had made it to second in line when the guy working the window took a break, with no one available to replace him. This was terrible planning by the track on its biggest day of the year. Lucky for me, the guys behind me held my place while I got my drink, which I stupidly left with my group (rookie mistake).
Maybe I missed the worst parts, but the infield to me seemed very jovial. I didn't see any fights, which is surprising given the amount of booze consumption. In addition to what I said above, there was one point where I accidently ran into a guy and half his beer spilled. I offered to buy him another one, but he said it was cool. Maybe I'm cynical, but this surprised me.
As it gets closer to the race, a few things happen. First is clothes start coming off. Unfortunately, this seems to be more with men, but for guys, there is plenty of scenery if you look around. Also, the ambulances start getting used more and more often, as more people don't make it to the end of the day.
The Derby is watched in front of video screens. It's loud, with everyone cheering of course. From where I was standing, I could see the horses go through turn one if I turned around. Potentially, I could have seen what happened to Eight Belles had I turned around right after the race. However, I was checking my tickets trying to find a winner (short answer: nope). We all found out about it a few minutes after the race.
After the Derby, the exodus out of there happens. At this point, the alcohol hit and everything became a blur. On the ride back to the hotel (which was in Cincinnati), all I remember is passing out and occasionally puking in a plastic bag.
In the end, do I recommend going at least once? Yes, unless you either are completely horse racing in every way (fair enough, especially now), or a recovering alcoholic. I'm already hoping I can go back next year.