Whisk the instant chocolate pudding mix with 3 cups cold milk until thickened, about 2 minutes.
Beat the softened cream cheese with powdered sugar until smooth and lump-free.
Fold the whipped topping (or whipped cream) and vanilla into the cream cheese mixture until fluffy.
Gently fold half of the prepared pudding into the cream cheese mixture until combined.
Reserve about 1/3 cup of crushed cookies for topping and set aside the rest for layers.
Spoon a layer of the pudding-cream mixture into serving glasses or a trifle dish.
Add a layer of crushed cookie crumbs over the cream layer to create the 'dirt' texture.
Add another layer of plain chocolate pudding, then repeat with the pudding-cream mixture.
Finish with a generous sprinkle of reserved cookie crumbs to resemble soil.
Arrange gummy worms on top, tucking a few slightly into the crumbs; chill at least 1 hour before serving.
Serve chilled and keep refrigerated; consume within 2–3 days for best texture.