Is there any way in Mac OS X 10.5.x to create a .dmg image from existing physical DVD, without a third-party tools installation? I have carefully examined "disk utility" and searched Google, but it seems that all people just write .dmg files, no information on how to create them from physical DVD disks.
Have you tried dd?
You can create an iso of the DVD in the following manner.
This should be what you want.
Disk utility is able to do that. Just select the disk you want an image from in the sidebar in Disk Utility and then click the "new image button" in the top icon bar.
Added note: Of course this doesn't work with CSS encoded video dvd's :)
Alternatively, open up Disk Utility - click on the DVD and then choose File -> Create New Disk Image from disk#s# (NAME_OF_DVD_HERE).
You can then save it somewhere (compressed or CD/DVD master) and you're good to go. (This does work with CSS'd DVDs)