That, my friend, sounds like bald faced hornets. They're no joke and comparable to European hornets. I worked pest control for a few years and I always dealt with those guys cautiously.
Yellow jackets are common to build nests below bushes or trees and will often have a guard near the pinky sized hole during daylight hours. If you choose to deal with them yourself, do so just after sunset, during night, or before sunrise. Some people drown them in scalding water or (if the nest is far enough away from the home) gasoline and light that ##### on fire. Pest control companies often use the product drione dust in the entry hole, which is a repellant. Not sure if it is available for public use, but if you get your hands on some use it liberally inside the entry tunnel. I've also treated yellow jackets that come in at holes near the peaks of houses and build nests in the insulation between a wall or ceiling and the drywall. They will chew through drywall and get into a home. If the drywall feels soggy or paper thin like it is water damaged, do NOT push through it! You will be attacked.
I worked for a small company out of Baltimore that charged $150 for non-existing customers and $100 for existing customers for yellow jackets, and $225 for bald faced or European hornets non-existing / $150 existing customers.
At least I keep seeing them crawl up into this tiny little gap in it.
