I've ordered a new gig-e switch and a 150 foot cat6 cable just in case. They will be here tomorrow and Friday respectively. I also have a 25 ft cat6 cable coming tomorrow (love Amazon)... so I should be able to rule out the closet switch and the interconnect with the other switches up here. That will either fix the problem or I'll end up snaking the 150 ft cable through the attic, down through the walls, and back to the basement. Always a good time.
I hope it's the switch or the closet->office cable.
The new switch seems to have fixed my dropped packets issues. The closet->office cable is currently limiting my max bandwidth but is still high enough not to matter in this case. The 25 ft cable I used for testing is not long enough to be a permanent solution in that case so it will wait. The issues with connecting to the server should be gone now, though.
Thank you, It's good to see it's fixed so I and the rest can stop complaining.