Today we commonly move data in the from one place to another using USB sticks. You can boot most PCs from a USB stick and that's become a popular way of infecting computers. But it wasn't always that way.
It's agreed that the first general purpose computer was ENIAC (Electronic Numerical Integrator And Computer) designed by John Mauchly and J. Presper Eckert of the University of Pennsylvania . ENIAC was programed using plugboard wiring and switches. If aa different program needed to be run the plugboard wires and switches had to be reconfigured - a lengthly and error prone process at best.
Eckert and Mauchly formed the Eckert Mauchly Computer Corporation (EMCC) to develop the EDVAC, ENIACS's successor. In 1947 EMCC contracted with the Northrop Corporation to build the BINAC (Binary Automatic Computer) - the first stored program computer built in the United States. BINAC conisted of two independent CPU. Each CPU had a 512 word acoustic mercury delay line memory unit for storing the program and data. The delay line memory was loaded from magnetic tape. In researching this article I was unable to find any information on the tape drives or the tape itself.
EMCC then produced the UNIVAC 1 UNIVERSAL Automatic Computer.) Manufactured by the Eckert Mauchly Computer Corporation for the U.S. Census Bureau it was delivered in March of 1951. As a general purpose computer it needed a method of reprogramming which allowed easily reloading the program being replaced. The answer was magnetic tape.
Information - either programs or data - could be written to a reel of tape either using a device known as a Unityper, transferred from punched cards or asa program's output data. The output data from one run could then be read, updated and written to a new tape the next time the program was re-run. The following YouTube video is courtesy of the Computer History Archives Project.