Then you aren't looking had enough...
When you've properly unlocked the visit, the tile will even be starred during the hours when the visit can happen, which should be between 8:30 and 12:30. If you aren't paying attention when talking to Serena (say, because you're trying to speed into her...