I would think you would end up using more energy in total when you turn it back on the next day. If the doors aren't being opened often, the fridge does not have to work very hard to keep the temperature within the set range assuming a reasonable ambient temperature. The compressor is not running for very long during each cycle, and the fans aren't going to draw much power. If the fridge is left off though, when you turn it back on, the compressor now has to run for longer to try and make up the temperature difference that built up while it was off.
This is the same reason that during the summer, it's suggested you increase the temperature of your thermostat during the day while you are at work, but not to turn off the system entirely as you end up spending more energy in total as the system works overtime to remove all the heat that built up.
Depending on how much room you have, you could freeze a few gallon jugs of water to place in the fridge. This will help to keep the temperature from rising just like ice in a cooler.