What are you saying "no" to?
Not only the thing that you don't want to do... but also the thing that you do want to do. The thing that does no harm, that's definitely better to have, that no one would say no to.
But that's not your priority. Not for now at least.
Unless you say no to it, this thing would distract you from the work that requires your focus, the work that matters.
You can only say "yes" to something when you say "no" to everything else.