I do have every reason to believe that in the episode Swing (season 10), at one point Fin baby-sat Eli while Kathy was visiting Kathleen in the hospital room. There was no scene of Fin and Eli together so it's alleged but to me, it makes perfect sense:
Fin nor Eli were seen at the hospital- they could've been in the waiting room. My best theory is that Fin was baby-sitting Eli while Elliot and Kathy were with Kathleen- he was there when Kathleen was admitted and I'm sure both Kathy and Elliot would trust Fin to care for their baby son. Truthfully? I would've loved to have seen Fin and Eli have a moment- it could've been a cute/sweet scene.