Java Active Remote Compiler (JARC)


Hello, I have attempted to build a remote compiler based on standard Java SE API [>1.6] http://java.sun.com/javase/6/docs/api/ and RMI, primarily to experiment with these technologies, and secondly to see if I could move the burden of CPU intensive operation of compilation away from the development environment or production hosts (in the case of dynamic compilation).  Pushing the compilation task across the network may reduce negative impact on other applications i.e. free up computation power for more important tasks, checking email, performing business logic and serving other clients for example.

Download
System Properties
Remote Server Configuration
Command line integration
Maven integration
Ant integration
Tomcat integration JSP compilation
JBoss Integration
Opencloud Rhino Slee Integration
INETD Integration
Compiler Compatibility
Future Research/Limitations
Support/Bug Reporting


Download jarc-all.jar jarc-client.jar jarc-server.jar jarc.sh jarc-deamon.sh rmid.policy jarc.policy
jarc-xinetd Download Sources jarc-sources.jar

a cool diagram

System Properties
  • jarc.home - MANDATORY - the base location as a URL ie http://www.w3.org/Addressing/
  • jarc.host - OPTIONAL DEFAULT localhost 127.0.0.1 ::1 - the target compilation host a comma seperated list of reachable servers running rmid and rmiregistry
  • jarc.port - OPTIONAL DEFAULT 2099 - the port of the service once activated in the rmid
  • java.rmi.server.codebase - OPTIONAL DEFAULT $jarc.home + version jar - if specified can be a URL or just a suffix and jarc.home is prepended
  • jarc.activation.security.policy - OPTIONAL DEFAULT $jarc.home + jarc.policy - if specified can be a URL or just a suffix and jarc.home is prepended
  • jarc.activation.file - OPTIONAL NO DEFAULT - the activation file - if specified can be a URL or just a suffix and jarc.home is prepended
  • jarc.debug - OPTIONAL DEFAULT false - print debug log tracers to standard out/err - also enabled with -verbose switch
  • rmid.port - OPTIONAL DEFAULT 1098 - the activation port if changed
N.B. System properties can be in the system environment uppercase with . replaced by _

To specify jarc.home
root@splice/> export JARC_HOME=file:///opt/jarc

To specify jarc.home
root@splice/> export JARC_HOST=beast

Remote Server Configuration

Normally just run the daemon on the target compilation platform, this launches the rmi registry and rmid activation daemon.
root@splice/> jarc-daemon.sh
root@splice/> kill-jarc.sh

To configure *Nix (Linux Solaris etc) servers without the need to launch jarc-daemon.sh on each server see INETD Integration section

Command line integration
At the moment JARC has shell scripts and batch files which launch Java, I have been looking into natively compiling the client side of the compiler.  Please let me know if you really need this.  Essentially it calls javac on the server so you can pass whatever arguments are available in the compiler.

Microsoft NOT YET
C:\> jarc.bat -Djarc.host=%HOST% -classpath %CLASSPATH% -d classes src\MyProject.java

*Nix Sh
root@splice/> jarc.sh -jarc.home=$JARC_HOME -jarc.host=$HOST -classpath $CLASSPATH -d classes src/MyProject.java

Maven integration
Easy to compile your existing maven applications with JARC.. just add the following tags to your compiler plugin configuration.  To make this portable across development teams see http://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-using-different-jdk.html and http://maven.apache.org/pom.html#Properties

<project>
    ...
    <properties>
        <jarc.home>http://www.pointdefence.net/jarc</jarc.home>
        <jarc.host>beast</jarc.host>
        <jarc.port>2009</jarc.port>
        <java.rmi.server.codebase>jarc-1.0-SNAPSHOT.jar<java.rmi.server.codebase>
        <jarc.activation.security.policy>${jarc.home}/jarc.policy<jarc.activation.security.policy>
    </properties>   
    <build>
       <plugins>
           <plugin>
               <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                <source>1.6</source>
                <target>1.6</target>
                <executable>/opt/jarc/bin/jarc.sh</executable>
                <fork>true</fork>
                <verbose>true</verbose>
            </configuration>
         </plugin>
    ...

To avoid hard-coding a filesystem path for the executable, you can use a property.
<executable>${build.compile}</executable>

Then define this property in settings.xml, or set an environment variable, so that the build remains portable

<settings>
...
<profiles>
...
<profile>
<id>compiler</id>
<properties>
<JAVA_1_4_HOME>C:\Program Files\Java\j2sdk1.4.2_09</JAVA_1_4_HOME>
</properties>
</profile>
</profiles>
...
<activeProfiles>
<activeProfile>compiler</activeProfile>
</activeProfiles>
</settings>

Ant integration

Copy or link the jarc-1.0.jar to $ANT_HOME/lib/ or $HOME/.ant/lib/

ln -s $JARC_HOME/jarc-1.0.jar $HOME/.ant/lib/

Set the build.compiler property on the comand line

ant -Dbuild.compiler=jarc.JarcAntCompiler

or in the build.xml, see http://ant.apache.org/manual/CoreTasks/property.html

<build>
<property name="build.compiler" value="jarc.JarcAntCompiler"/>
<property name="jarc.home" value="http://www.pointdefence.net/jarc"/>
<property name="jarc.host" value="::1"/>
<property name="jarc.port" value="2009"/>
<property name="java.rmi.server.codebase" value="${jarc.home}/jarc-1.0-SNAPSHOT.jar"/>
<property name="jarc.activation.security.policy" value="${jarc.home}/jarc.policy"/>
...

L
ook here for more information about the ant compilation task http://ant.apache.org/manual/CoreTasks/javac.html and here http://ant.apache.org/manual/running.html#libs

Tomcat integration JSP compilation

When JSPs are requested they are compiled on demand. JARC distributes as a web application so you can access it in your network.

ln -s $JARC_HOME/jarc-1.0.jar $TOMCAT_HOME/common/lib/

To configure tomcat globally edit the tomcat/conf/web.xml or just edit the WEB-INF/web.xml of your application.  See the web.xml of the jarc-1.0.war file for example

<web-app>
     ...
    
<env-entry>
           <env-entry-name>jarc.host</env-entry-name>
           <env-entry-value>beast</env-entry-value>
           <env-entry-type>java.lang.String</env-entry-type>
     </env-entry>
    
<env-entry>
            <env-entry-name>jarc.home</env-entry-name>
           <env-entry-value>http://beast/jarc/</env-entry-value>
           <env-entry-type>java.lang.String</env-entry-type>
     </env-entry>
    
<env-entry>
           <env-entry-name>jarc.port</env-entry-name>
           <env-entry-value>2009</env-entry-value>
           <env-entry-type>java.lang.String</env-entry-type>
     </env-entry>
     <env-entry>
           <env-entry-name>java.rmi.server.codebase</env-entry-name>
           <env-entry-value>jarc-1.0-SNAPSHOT.jar</env-entry-value>
           <env-entry-type>java.lang.String</env-entry-type>
     </env-entry>
     <env-entry>
           <env-entry-name>jarc.activation.security.policy</env-entry-name>
           <env-entry-value>jarc.policy</env-entry-value>
           <env-entry-type>java.lang.String</env-entry-type>
     </env-entry>
     ...
<servlet>
       <servlet-name>jsp</servlet-name>
       <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
       ...
       <init-param>
           <param-name>compiler</param-name>
           <param-value>jarc.JarcAntCompiler</param-value>
       </init-param>
       ...
</servlet>
...

For more information about tomcat and jasper see http://tomcat.apache.org/tomcat-6.0-doc/jasper-howto.html#Production%20Configuration

JBoss Integration
Not yet sorted

Opencloud Rhino SLEE Integration

To configure Rhino edit the $RHINO_BASE/conf/jvm-args and add the following line.

-Dcom.opencloud.javac.external=@RHINO_BASE@/jarc.sh

Then copy the jarc.sh compiler to the $RHINO_BASE directory.

INETD Integration
To run the JARC server on machine without running jarc-daemon.sh  you can setup inetd to start the rmid automagically when JARC client demands. See
http://java.sun.com/javase/6/docs/technotes/guides/rmi/inetd/rmid-inetd.html for more information.

Add the following entries to /etc/inetd.conf for some reason unbeknownst to myself inetd loops and hangs with rmid.  If this happens on your inetd impl then simply run rmid seperatley.

rmid stream tcp4 wait root /java/bin/rmid rmid -Djava.io.tmpdir=/tmp \
 -J-Djava.security.policy=file:/jarc/bin/rmid.policy -J-Djava.net.preferIPv4Stack=true\
 -C
-Djava.net.preferIPv4Stack=true -log /var/log/rmid

jarc stream tcp4 wait root /opt/java/bin/java java -Djarc.inetd -Djava.net.preferIPv4Stack=true \
 -Djava.net.preferIPv6Addresses=false -Djava.rmi.server.codebase=file:/jarc/bin/jarc-1.0-SNAPSHOT.jar \
 -Djarc.activation.security.policy=file:/jarc/bin/jarc.policy -jar /jarc/bin/jarc-1.0-SNAPSHOT.jar \
 -log /var/log/jarc.log


service rmid 
{
    socket_type = stream  
    protocol    = tcp    
    port        = 1098
    wait        = yes
    user        = root 
    server      = /root/jdk1.7.0_06/bin/rmid 
    server_args = -C-Djava.util.logging.manager -C-Djava.util.logging.config.file=/root/jarc/logging.properties \
    -C-Djava.security.policy=file:/root/jarc/jarc.policy -C-Djava.rmi.server.logCalls=true -J-Djava.rmi.server.logCalls=true -J-Djava.io.tmpdir=/tmp \
    -J-Djava.security.policy=file:/root/jarc/rmid.policy -J-Djava.net.preferIPv4Stack=true -C-Djava.net.preferIPv4Stack=true -log /var/log/rmid 
}
                                            

service jarc 
{   
    socket_type = stream  
    protocol    = tcp
    port        = 2009
    wait        = yes
    user        = root  
    server      = /root/jdk1.7.0_06/bin/java
    server_args = -Djava.util.logging.manager -Djava.util.logging.config.file=/root/jarc/logging.properties \
      -Djava.rmi.server.disableHttp=true -Dsun.rmi.transport.tcp.handshakeTimeout=1000 -Djava.rmi.server.hostname=10.0.8.1 \
      -Djarc.inetd -Djava.rmi.server.logCalls=true -Djava.net.preferIPv4Stack=true  -Djava.rmi.server.codebase=http://10.0.8.1:8500/jarc-www/jarc-1.0-SNAPSHOT-server.jar -Djarc.debug=true -Djarc.activation.security.policy=file:/root/jarc/jarc.policy \
       -jar /root/jarc/jarc-1.0-SNAPSHOT-server.jar -log /var/log/jarc.log 
    #-D-agentlib:jdwp=transport=dt_socket,address=localhost:9998,server=y,suspend=y -Djava.net.preferIPv6Addresses=false
}

And add the entry to /etc/services
rmid 1098/tcp
jarc 2009/tcp

Compiler Compatibility

SUN Javac
This is the only one I have tested on and it seems to mostly work.


GNU Javac ??


Future Research/Limitations
  • Round robin or distributed compilation server targets
  • Take metrics from network utilization vs cpu time to see if this project is actually saving time!
  • Nativley compile client so command line launch times are faster.
  • Jetty integration
  • Jboss integration
  • Make errors look like javac errors
  • Look at server threading and multi core cpu utilization

Support/Bug Reporting

    Java system properties
  • java.io.tmpdir=/tmp
  • java.net.preferIPv4Stack=true
  • java.net.preferIPv6Addresses=false
http://docs.oracle.com/javase/6/docs/technotes/guides/rmi/sunrmiproperties.html http://docs.oracle.com/javase/6/docs/technotes/guides/rmi/javarmiproperties.html For more information or support just email me (jarc pointdefence.net) call me or stop me on the street and ask =)