Scale up is about vertical scalability. This means increasing the capacity by adding more CPU and memory.
Scale out is about horizontal scalability. This means increasing the number of servers available.
Brent Ozar wrote a couple of really interesting articles
Scaling up or scaling out?
Scaling up of scaling out? part 2
Interesting analogy here
Scale Up or Scale Out your Data Problems? A Space Analogy