Greyhound bus line provides numerous service routes to surrounding states and other major cities throughout the U.S. The Greyhound busses depart from the Port Authority Bus Terminal at 8th Avenue and 41st Street near Times Square.
Regional bus companies service many routes between New York City and other cities in the surrounding northeastern states. They are often called "the Chinese Buses" since many of them operate out of Chinatown. They provide inexpensive transportation to Boston, Philadelphia, Washington DC, Atlantic City and many other destinations.
New York City is one of a few cities in the USA that has adequate train service.
Much of Long Island is accessible via the Long Island Railroad departing from Penn Station. Here is the Long Island Railroad map: Long Island Railroad map
Port Authority Trans-Hudson Corporation (PATH) operates as the main transit that links Manhattan from Penn Station or the World Trade Centers Terminal with Newark and Hoboken. Here is a link to the PATH map: PATH Map
Amtrak, the national passenger railroad, connects New York with many major U.S. cites. It provides direct routes to Chicago, New Orleans, and Washington D.C. The AMTRAK trains all depart from Pennsylvania Station at 8th Avenue and 31st Street.