The DITA Open Toolkit User Guide devotes a lot of attention to Windows
and Linux but gives Mac OS X scant attention. The brief Mac installation section on page 58 simply says

Before installing DITA Open Toolkit and its prerequisite software on Mac OS X, check to see if any of the tools are already installed on your system and, if so, whether the version you have is supported (see System requirements and supported applications on page 47). Java is a core component of Mac OS X. Newer versions of Mac OS X include the full version of the Java JDK 1.4.2 by default. This version of the JDK includes Ant
and the Xalan-J XSLT processor, as well. Other tools you want to install may be included on the Mac OS X Developer's Tools on the product DVD.

To install the Toolkit, extract the zip file to your HOME directory, then edit your login rc file to include the Toolkit in your CLASSPATH.

You must use a Terminal window to enter commands on the command line.

If you have not changed your shell, enter this (you should be logged in as an account with adminsistrator privileges):

export CLASSPATH="$HOME/DITA-OT/lib:$HOME/DITA-OT/lib/dost.jar"

The .profile file and others created are "hidden" files. It's important to remember that Mac OS X is a BSD UNIX-based OS. All filenames that have a period before them are hidden under normal circumstances. However you can view these files in the Terminal using the ls -a option.

An easier method to install the OT on Mac is the following:

The DITA-OT "Full" aka "Easy" package has a "startcmd.sh" command,
which purports to work on Mac OS X as well as on Linux. You can
source it with something like these two commands:

export DITA_HOME="$HOME/DITA-OT"
"$DITA_HOME/startcmd.sh"

Other notes:

There's a Java Preferences application in the Java folder under
Utilities. You can use this to set which J2SE version to run.

Automator and AppleScript can run shell commands. Once you've figured
out what you need to run in Terminal, you should be able to transfer
these to either Automator or AppleScript, and add appropriate variables.