Issues during migration of platform to OSGi

1. Certain framework classes had to be part of boot delegation like Terracotta.

Added com.tc.* to org.osgi.framework.bootdelegation = \ in

/home/pg/thirdparty/springsource-dm-kernel-2.0.0.RELEASE/lib/java6-server.profile

2.Certain jboss imports were not getting part of package imports when jars were run through bundlor

So had to explicitly add these to bundles which required JMS(JBoss MQ) functionalities.

org.jboss.logging,
org.jboss.mq.referenceable,
org.jnp.interfaces,
org.jboss.util,


3. If there is any Custom Class Loader created in the code ,special care has to be taken to ensure that you set the custom class loaders parent to Bundle Classloader.


4. Special care needs to be taken that classes loaded through custom class loader should see all its imports and exports through its parent Bundles manifest.


5. Where ever class was loaded using Class.forname() we had to change it to ensure these where done through Thread Context Classloader where classes were laoded from other bundle.

6. Since we use jbossallclient.jar for JMS(Jboss MQ) we had issues as this was conflicting with packages in bootdelegation.

We could either solve this using explicit imports from system bundle(bundle 0).

a.

BootClasspath was exporting javax.management

Removed javax.management from jboss all client jar - Using bundlor excluded exports manifest

b.

Because jboss-allclient.jar also exports javax.jms which is also exported by javax.jms_1.1.0 bundle

Ensured jbossallclient.jar imports this and doesnt export it by changing manifest.


7.Since there were multiple bundles exporting org.apache.log4j , i had to explicitly mention bundle
symbolic names for the imports.

There were certain cases of cyclic dependency since it was legacy code.

Code in one bundle was initialized(spring application context) by code in another bundle. This cyclic dependency was solved by
explicitly marking this imports as optional(resolution="optional").


8. Wherever Application Context was read using regular expression had to change classpath: to classpath*: to ensure that all classpath resources are read.

BootStrap Initialization GuideLines (For same code base to work in both OSGi and Non-OSGi environment)

During the migration of the platform into OSGi we had use cases where we needed the platform to work in both OSGi and non OSGi cases until the OSGi platform had stabilized.But our mail goal was we needed the same code based and packaging to work in both OSGi and non OSGi use cases.

We had come up with list of guidelines and conventions to achieve the same.

One major assumption was that in a non OSGi case there has to be a single application Context as against OSGi where there is application context
per bundle.


1. Separate xml files for OSGi service definitions and bean definitions.

2. Naming convention for spring xml files are as follows

<MODULE_NAME>-osgi-context.xml
<MODULE_NAME>-bean-context.xml
3.Naming convention for beans to be exported as services

Service Exposing side
<bean id="<MODULE_NAME>service" class="XXX.YYY.ZZZ"/>
<osgi:service ref=""<MODULE_NAME>service"" interface="bla.bla.IService"/>

Service Reference side
<osgi:ref interface=" bla.bla.IService" id="<MODULE_NAME>service"/>
Now this <MODULE_NAME>service can be used as regular bean.
This convention will help us run the same code base both in OSGi and non OSGi cases without changing these xmls. Note that in non OSgi cases we will initialize META-INF/spring/*-bean-context.xml. This will help in unit testing so that only required xmls can be initialized.

4. All spring bean definition files have to be in the following location - META-INF/spring/*

5. Ensure that in NON-OSGI cases individual products/services/modules do not independently initialize the application Context.There should be a independent initialization moduel which initilialises the app context across all products(jars) in the Vm based on some naming convention followed by the products.This will help easy wiring of beans across products which in other cases are done through OSGi services.Also the common module will expose a method in for accessing the beans present in the context.

6. Conventions have to be followed so that there are no same named beans across products as there is a single application context and objects will get overridden.

7.Boot Strap initialized xmls with this regular expression - [*-bean-context.xml]

[** Ensure that if any module in not following all these conventions the spring xml file has to be in any other folder than META-INF/spring ***[so that DM server doesnt initialize it ]* . *Ensure that in any case the spring xmls** are not initialized twice *[all the above guidelines are just to ensure that ] ***]

8. Property File initialization

Property placeholder config definitions containing files should be seperated from bean and osgi definition files and should be present in META-INF/spring folder.

Naming convention pgconfig-<MODULE_NAME>.properties

This will be initialized in bootstrap based on this expression in teh bootstrap propertyPlaceHolconfigurator : "classpath*:pgconfig*.properties"

{"classpath*:META-INF/spring/bootstrapconfig.xml","classpath*:/META-INF/spring/*-bean-context.xml"};

bootstrapconfig.xml contains --> <context:property-placeholder location="classpath*:pgconfig*.properties"/>

9.Avoid using Bundle Activator and instead use spring config (init method) for initialization as this will give us a common approach in both OSGi and Non OSGi cases.

OSGi Guidelines

1.Interaction between non-core bundles which can be hot-deployed should be through services (Interfaces).

2.Bundle should Import packages only for utilities, datatypes packages / classes and interfaces.

3.Interface and implementation should be separately bundled. The datatypes associated with service contracts can be bundled along with the interfaces.

4.All third party jars should be converted into bundles. Avoid using "external" in Bundle-classpath.

5.Avoid split package. Proper package structure should be followed. Two bundles should not have same package structure. We can have package structure like -.

6.TC shared classes to be bundled into separate bundles.

7.Avoid static variables and blocks.

8.Avoid Class.forname instead use Classloader.loadclass(). Class.forname may cause issues when bundle is hot deployed as initiating classloader caches the loaded class.

9.Classes present in separate dependent bundles should be loaded using TCCL. For example in jdbcframework all TYPE classes would be bundled with the respective bundle. These classes have to be loaded by jdbcframework bundle. Since classes present in respective bundle will not be visible to jdbcframewok bundle, so jdbcframework bundle has to use TCCL to load these classes.

10.Cyclic dependency should not exist between bundles.

11.Resource clean up - Each bundle / bean should clean up its resources like threads, sockets, cache, file handles, connections etc using its life cycle method (using "destroy" of bean or "stop" of "bundle activator")

12.Use the bean initialization method to initialize the properties and avoid putting initialization logic into constructor.

13.Native library to be loaded once. The code that is loading the native library should be part of core bundle which should not be hot deployed.

14.Prefer composition over inheritance.

15.Use AOP for common concerns - prefer compile time weaving over load-time / runtime weaving.

16.Use DM kernel when web container is not required.

17.Prefer import package over import-bundle.

18.All In-House jars should not have resolution optional for package imports.[Just to ensure that in production enviroment dependencies are resolved at deployment time rather than run time.]

19.All third party converted jars have resolution=optional.Else this pulled a not of other jars which were never used.

20.All log4j imports should have bundle symbolic name for lo4j to resolve properly to log4j instead of slf4j - something like

org.apache.log4j.*;bundle-symbolic-name="com.springsource.org.apache.log4j"

This can be added as part of Bundlor template during building the jar.

21.Manifest is generated at build time based on template(Spring Utility like Bundlor can be used for the same).

Thread Context Classloader in Deserialization - ObjectInputStream

We had uses cases, especially with OSGI where we had to use TCCL in deserialization of Object InputStream.

Sample method

public static Object getObject(byte[] bArr){
try{
ByteArrayInputStream bais = new ByteArrayInputStream(bArr);
ObjectInputStream ois = new ObjectInputStream(bais);
return ois.readObject();
}catch(Exception err){
err.printStackTrace();
}
return null;
}


return ois.readObject() - fails because ObjectInputStream doesnt take into consideration TCCL when deserializing.

In OSGi we need to deserialize this using TCCL because objects being deserialized are in another bundle and framework bundle canot see these classes.

Java has provided a way to solve this . We need to extend ObjectInputStream and overide the
public Class resolveClass(ObjectStreamClass desc) method.

[Since resolveClass class is protected method in ObjectInputStream ]

Code


public class MyObjectInputStream extends ObjectInputStream {

@Override
public Class resolveClass(ObjectStreamClass desc) throws IOException,
ClassNotFoundException {
ClassLoader currentTccl = null;
try {
currentTccl = Thread.currentThread().getContextClassLoader();
return currentTccl.loadClass(desc.getName());
} catch (Exception e) {
}

}
return super.resolveClass(desc);
}

public MyObjectInputStream(InputStream in) throws IOException {
super(in);
}

}

Now we can use MyObjectInputStream which ensures that TCCL will have visibility to deserialize the class.

MyObjectInputStream ois = new MyObjectInputStream(bais);
return ois.readObject();

Using Bundlor for automated generation of Osgi manifest files

One of the most useful utilities which helps in automated creation of manifest files. This gives all the hooks required for customization.

A sample template file used for generation

Bundle-Name: ${bundle.name}
Bundle-Description: ${bundle.description}
Bundle-ManifestVersion: 2
Bundle-SymbolicName: ${bundle.symbolic.name}
Bundle-Vendor: ${bundle.vendor}
Import-Template: org.apache.log4j.*;bundle-symbolic-name="com.springsource.org.apache.log4j", *
Excluded-Imports: com.sun.*,javax.xml.*,org.apache.xerces.jaxp.*,org.w3c.*,org.xml.*,sun.*,com.tc.*


Here the Import template functionality gives complete flexibility to control my imports> This ensures that all subpackages of org.apache.lo4j if imported by bundle will have bundle symbolic names attached to it.

If there is any private package , based on some naming convention we can exclude this from being exported. Also boot delegated packages should be excluded for exporting.

Boot delegated package exports can be avoided by passing osgi profile file to bundlor.
At this moment - Bundlor 1.0.0 had a Bug with this feature which is fixed in the daily snapshot version.
Ref : http://forum.springsource.org/showthread.php?t=87312

Hence bundlor automates a lot of these things and there is no need for static commit of manifest files.

This can be easily integrated with any build for the project as follows

<target name="bundlor.init">
<taskdef resource="com/springsource/bundlor/ant/antlib.xml"
uri="antlib:com.springsource.bundlor.ant">
<classpath id="bundlor.classpath">
<fileset dir="${bundlor.home}/dist"/>
<fileset dir="${bundlor.home}/lib"/>
</classpath>
</taskdef>
</target>


<bundlor:bundlor
inputPath="${basedir}/xxx_jar.jar"
outputPath="${basedir}/xxx_bundle.jar"
manifestTemplatePath="${basedir}/${bundlor.config.dir}/xxx.mf"
propertiesPath="${basedir}/${bundlor.config.dir}/xxx.properties"
osgiProfilePath="${basedir}/${bundlor.home}/profile/java6-server.profile"/>

Ref: http://www.springsource.org/bundlor

Integration of Terracotta with Spring Dm server

At this moment I am using the following version of the softwares

Spring DM server(or DM Kernel) - 2.0.1 version
Terracotta- version 3.2.1

Steps

In Dm server

1. Modify springsource-dm-kernel-2.0.1.RELEASE\bin\dmk.sh to start with terracotta cleint.

Change $JAVA_HOME/bin/java to $TERRACOTTA_CLIENT_PATH/bin/dso-java.sh

2. Change springsource-dm-kernel-2.0.1/lib/java6-server.profile to include terracotta classes in boot delegation path

Add under org.osgi.framework.bootdelegation = \
com.tc.*,\
com.tcclient,\
com.tcclient.*

Changes in Terracotta

1. change tc-config.xml to have the following tims

<modules>
<module name="tim-equinox-3.5.1" version="1.1.1"/>
<module name="tim-ehcache-2.0" version="1.5.2"/>
<module name="tim-distributed-cache" version="1.3.2"/>
<module name="tim-concurrent-collections" version="1.3.2"/>
</modules>

2. Add the following app group element in tc-config.xml

<app-group name="osgiAppGroup">
<named-classloader>Standard.system</named-classloader>
<named-classloader>dso-shared.0.0.0</named-classloader>
</app-group>


Thw app group element in only required if you want to share dso classes between standalone JVM and Spring Dm server(where dso classes are laoded by bundle classlaoder).

Also important thing to be noted is that inside a JVM all shared classes(dso's) have to be laoded by a single classloader.[Inside spring Dm server all dso classes should be a single bundle]

Soft Reference /Weak Refernce /Phantom Reference / finalize()

Using teh above types we get a refernce to a object which ensure that teh object can still be GCed.

Main purpose of all the above types is to ensure that the reference is not held and hence GC can
collect all these objects from the heap.

The order in which the reference - from Strong to Weak are as follows
Strong reference,Soft Reference ,Weak reference and Phantom Reference

Strong Reference - are normal references which will not be garbage collected because some objects will have reference to it.

SoftReference -
Having a soft Reference to a object will not stop GC on that object. By using SoftRefernce the virtual machines does not clear this references until there is shortage of memory.But this ensures that before the system crashes with OutOfMemory errors all soft references will be GC'ed.

Hence this becomes a very good candidate for maintaing cache.Thus objects will remain in cache until there is enough memory. Once there is shortage they will be collected.


Weak Reference - Is similar to soft reference with the only difference that it will be GC'ed as soon as there are no other references . This will be very handly in most cases where there exists explicit refernce to objects which stop garbage colection.

If Weak Reference is not used , we need to explicitly clear the contents as part of cleanup.

WeakHashMap is one such data structure which will automatically clear references when there are not other references to objects. WeakHashMap works exactly like HashMap, except that the keys (not the values!) are referred to using weak references .If a WeakHashMap key becomes garbage, its entry is removed automatically.


Both weak and Softreference have constructors of format

WeakReference(Object referent, ReferenceQueue q) & WeakReference(Object referent)
SoftReference(Object referent, ReferenceQueue q) & SoftReference(Object referent)

where ReferenceQueue is queue which will be populated with objects(Weak/Soft references) which are GC'ed.[The objects they are pointing to will be GC'ed after which these references become dead]

Hence we can use this ReferenceQueue info for cleanup purposes.


Reference queues

Once a WeakReference starts returning null, the object it pointed to has become garbage and the WeakReference object is pretty much useless. This generally means that some sort of cleanup is required;

The ReferenceQueue class makes it easy to keep track of dead references. If you pass a ReferenceQueue into a weak reference's constructor, the reference object will be automatically inserted into the reference queue when the object to which it pointed becomes garbage. You can then, at some regular interval, process the ReferenceQueue and perform whatever cleanup is needed for dead references.

The remove() method can be used to get notifications on objects Garbage Collected.
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/ref/ReferenceQueue.html

To access WeakReference or SoftReference we have to use the public Object get().
If this reference object has been cleared, either by the program or by the garbage collector, then this method returns null.


A soft reference is exactly like a weak reference, except that it is less eager to throw away the object to which it refers. An object which is only weakly reachable (the strongest references to it are WeakReferences) will be discarded at the next garbage collection cycle, but an object which is softly reachable will generally stick around for a while


Finalize()


This is Called by the garbage collector on an object when garbage collection determines that there are no more references to the object.

This has to be carefully used because sometime improper use of this method can get teh object back to life.

This is also costly because garbage collection in its every traversal has to keep account of the finalize method variables and exclude it from considering as normal reference.Also until finalize method is called other objects will not be GC'ed because of its changes to come back to life. This can be solved with Phantom reference.

Finalize() method is costly -hence phantom reference can be used for the same purpose.

Phantom Reference

Phantom references are most often used for scheduling pre-mortem cleanup actions in a more flexible way than is possible with the Java finalization mechanism.

In order to ensure that a reclaimable object remains so, the referent of a phantom reference may not be retrieved: The get method of a phantom reference always returns null.

A phantom reference is quite different than either SoftReference or WeakReference. Its grip on its object is so tenuous that you can't even retrieve the object -- its get() method always returns null.

The only use for such a reference is keeping track of when it gets enqueued into a ReferenceQueue, as at that point you know the object to which it pointed is dead and perform some cleanup.

Difference b/ Weak-Soft & Phantom Reference

The difference is in exactly when the enqueuing happens. Weak/Soft References are enqueued as soon as the object to which they point becomes weakly reachable. This is before finalization or garbage collection has actually happened; in theory the object could even be "resurrected" by an unorthodox finalize() method, but the WeakReference would remain dead.

PhantomReferences are enqueued only when the object is physically removed from memory, and the get() method always returns null specifically to prevent you from being able to "resurrect" an almost-dead object.

2 Usecases with PhantomReference

1. Only way to determine exactly when an object was removed from memory based on which some actions can be performed

2.PhantomReferences avoid a fundamental problem with finalization: finalize() methods can "resurrect" objects by creating new strong references to them. Hence garbage collection of teh object is postponed untill finalize metdod is called.(because finaize object may
reconstruct the object ).With phantom reference no way the objects can be reconstructed and Gc will happen soon


Reference.
http://weblogs.java.net/blog/enicholas/archive/2006/05/understanding_w.html
http://mindprod.com/jgloss/weak.html
http://www.javalobby.org/forums/thread.jspa?threadID=16520&messageID=91822406&tstart=0
http://onjava.com/pub/a/onjava/2001/07/09/optimization.html?page=2