If you want the refrigerator to run when you are not on shore power, the inverter must be ON.
If you are plugged into shore power, you may want to leave the inverter ON anyway because if there is a power outage, inverter power from the batteries will automatically take over and keep the refrigerator running until power is restored.