You don’t have to have it. Part D is optional, and nobody will make you sign up.
What catches people is the penalty for waiting. If you go about two months or more without drug coverage after you’re first eligible, Medicare adds a charge when you do sign up. The charge grows a little for every month you went without, and it’s added to your premium each month. It lasts the whole time you have drug coverage. A year or two of waiting can turn into a noticeable amount over a long retirement.
So for most healthy people the math favors a low-cost plan now. You pay a small premium for something you may not use, and the penalty never starts. You’re also covered if a new prescription shows up in the middle of the year.
When you can skip it safely. If you already have drug coverage that Medicare counts as good enough, there’s no penalty. A plan from an employer or a former employer often qualifies, and so do VA and TRICARE benefits. Keep the letter that says so.
If you’re in Medicare Advantage. Most of those plans already include drug coverage, so there’s nothing extra to buy.