Matlab is something that I don't use, but I know we have a few Matlab users in the forums here. Here is what I would suggest. Start with the oil, oil used to cool transformers must be non conducting like pure mineral oil. Next start looking at actual designs of an oil cooled transformer as to reservoir for the oil and note how the oil actually moves using convection (some using pumps). You need the coolant (oil) and a way to have it move (convection or pump). Remember in the design that an oil filled transformer needs a relief valve for emergency pressure relief. Just think about and list all the important characteristics involved in the design. Sorry I can't help with Matlab unfortunately. Hopefully a Matlab user will see this thread and contribute to it. Additionally some large transformers use cooling fans with fins to cool the oil as it removes heat.
Ron