Just did another test, when your vCenter is joined to the same domain as the principal's domain, you apparently need to use the NetBIOS format.
$folderName='PermissionTest'
$roleName='Minimal'
$group='LOCAL\testgroup'
$folder=Get-Folder-Name $folderName
$role=Get-VIRole-Name $roleName
New-VIPermission-Entity $folder-Principal $group-Role $role-Confirm:$false
