Here are three ways to create compressed .dmg files on MacOS X:
Create a disk image with Disk Utility and compress it later
This is the way I originally did this:
Start Disk Utility
Click the “New Image” button
Choose how large you want the image
The image will automatically be mounted and visible in the Finder
Drag files to it
Go back to Disk Utility and click “Convert”
Select the image you created earlier; the default settings will compress this image
Enter a new name for the compressed image
Eject the uncompressed image in the Finder and remove it

This works, but it’s a bit more complicated than I like. So here’s an easier way:
Create a compressed disk image without intermediate steps
This is a lot easier than the first method, but most people are unaware of this option.
Create a folder containing all the files you want to add to the disk image
Start Disk Utility
Choose File -> New -> Disk Image from Folder from the menu
Select the folder containing your files
Enter the name of your disk image
Disk Utility will now create a compressed image containing the files in the folder you created in the first step.

Using the CLI
And of course there is an even easier way: use the command line. The hdiutil command can perform all the tasks above; here’s a simple script that gives an example of what you can do with it:
echo "Creating uncompressed disk image:"
hdiutil create -size $SIZE -fs HFS+J -volname $IMAGENAME \
echo "Mounting uncompressed disk image:"
hdiutil attach $IMAGENAME-temp.dmg -readwrite -mount required
echo "Image mounted. Drag files to it and press enter when ready. "
echo "Unmounting uncompressed image"
hdiutil detach /Volumes/$IMAGENAME/
echo "Creating compressed image"
hdiutil convert $IMAGENAME-temp.dmg -format UDZO \
-imagekey zlib-level=9 -o $IMAGENAME.dmg
echo "Removing temporary image"
rm $IMAGENAME-temp.dmg
echo "Done"