Say no to the request, not the person.
You’re not rejecting the person, just declining his invitation. So make that clear. Let him know what you respect about him — maybe you admire the work he’s doing, or recognize his passion or generosity. Maybe you would love to meet for lunch. Don’t fake this — even if you don’t like the person making the request, simply being polite and kind will communicate that you aren’t rejecting him.