This museum is sited above the Rotherhithe Tunnel built by Brunel in the 19th century. It’s an interesting site, but you should manage your expectations before visiting. The museum (or rather, mini-museum) is housed in an original engine house above the tunnel. The pump engines are no longer there, but you can see a video and display panels about Brunel (father and son) and their project. There is access by guided tour to the original access chamber for the tunnel, a large underground cylindrical space. You have to climb over a small wall, go through a 4ft high opening and descend a scaffold tower. (Definitely no disabled access!) There is a modern concrete floor between you and the train tunnels. There is also a pleasant garden sited above the chamber roof.
There is no access to the under-river tunnel from the Museum. Instead, you have three options:
1) Take a London Overground train through the tunnel at any time of your convenience. You may not see much. Most of the Victorian brickwork is now coated with shot-crete.
2) Take a train excursion at a time when the tunnel lights are turned on.
3) Book for one of the rare walks through the tunnel, when the trains have been stopped for maintenance. It will cost you around £18.
By public transport, you can reach the museum by London Overground train to Rotherhithe, or by tube to Bermonsey, then bus. There is a great view of the river a few yards to the North.