Anyone giving their baby almond milk?

So, I’m looking to wean William from his two after nap nursing sessions. I have a lot of lactose intolerance issues (both myself and my family). Our doctor has recommended just giving him formula for those two feeds but it doesn’t make sense to me… I’ll wean him onto formula only to switch in a few months? Should I just give him lactose free milk? How much? Any advice would be great!!