Alternative solution:
// example values
$width = 250;
$height = 250;
$cornerRadius = 10;
// create mask image
$mask = new Imagick();
$mask->newImage($width, $height, new ImagickPixel('transparent'), 'png');
// create the rounded rectangle
$shape = new ImagickDraw();
$shape->setFillColor(new ImagickPixel('black'));
$shape->roundRectangle(0, 0, $width, $height, $cornerRadius, $cornerRadius);
// draw the rectangle
$mask->drawImage($shape);
// apply mask
$image->compositeImage($mask, Imagick::COMPOSITE_DSTIN, 0, 0);
I found solution here:
https://github.com/Imagick/imagick/issues/213#issuecomment-385928740