I've never seen any trouble at Millwall, been 3 or 4 times. To be honest its the safest away trip.... You get on the overground at London Bridge and its the first stop (2 mins). You get off the train and there is a walkway straight to the away turnstile.
Overground is the easiest as someone said earlier there is a walkway direct to the away end. However, there are two tube stations about 10 & 15 mins walk (Surrey Quays & New Cross Gate)