Actually, the 'Bandit Leader' boss mob drops the 'Brooch' item that you turn in to Gloisin, he doesn't spawn down with the other mobs, but if you'll go up a level and around above the bandit camp, there's a small tower ruin with two chests, he spawns there.