I'm twice his age, but I'm terribly infatuated with him.?

I'm 40something and he's 21. We work together but have never been intimate. I'm extremely infatuated, an would like to try it out, but co-peers find it distasteful. Family tells me that I would be **** on by a younger man, and co-peers say, I'm chester the molester. Hmmm. If a man, no matter what age finds me attractive, and if I find a guy attractive should I be concerned with age as long as he's an adult? Is age just a number or does it mean alot more than that? No I couldn't provide him children in the future, but should I persue his offer to come visit him at his apartment. Other concern is I haven't been ually active in over 3yrs. Layin dormant I guess, but he seems to strike a nerve. What should I do? Should I forget about my infatuation. I'm black and he's white. He also is talking to a girl in his own age group, but what does that mean? Does it mean that he's doing what society want?