Elevators can use 20-100 kWh per day.* The amount of power used is greater for going up than down, and increases with more passengers carried. Starting from a stop requires a burst of energy, so making fewer stops uses less electricity. Walking, on the other hand, uses no electricity and gives you a chance throughout the day to get some healthy exercise. Choose to take the stairs, particularly if you are only going a couple of floors and do not have a cart or are carrying items that would otherwise require using the elevator.
* R. S. Caporale , Space and Physical Requirements in Strakosch, G.R., and R. S. Caporale, eds. 2010. The Vertical Transportation Handbook. 4th ed. John Wiley &Sons, Inc., NJ