When the hard drive, for example, has to read pieces of data from multiple different areas on the drive, it can not access the whole of the data as fast as it could if it had all been written together in the same area of the drive.
As an analogy, imagine that you want to play a card game that requires an entire deck of cards. Before you can play the game, you have to retrieve the deck from wherever it might be. If the cards are spread all over a room, the time needed to gather them together and put them in order would be much greater than if they were sitting on the table, nicely organized.
A deck of cards spread all over a room can be thought of as a fragmented deck of cards, much like the fragmented data on a hard drive that when gathered together, might equal a file you're after or a particular software program.
Defragmentation then would involve gathering together and organizing data, or in the deck of cards analogy, gathering together and sorting the playing cards.
Defragmenting the files on a hard drive is usually an effective way of speeding up access to those files. See my list of Free Defrag Programs for updated reviews of programs that will do this.
Examples: "My computer was running very slow so I ran a quick scan with my defrag software to see how fragmented my drive was. The scan said my hard drive was significantly fragmented and that performing a defragmentation would really help speed it up."