Coming back...

Haven't written to this blog in a while - getting caught up with getting the project I have been working on working, but with some of the issues we are finding and the solutions to those issues, I really need to document them for future reference.

I have also been getting deeper into iOS and Android development, and have figured out some cool things there to start documenting and keeping around.

So major posts coming regarding Java EE 6/OSGi hybrid apps, iOS and Android development (as well as anything else I want to blab about just to keep record).

If others can find some of the snippets to help like I did searching around - would be great.

mysql error: Got a packet bigger than 'max_allowed_packet' bytes

I had recently been working on a project where I was getting db dumps (using mysqldump) - and having issues importing the db back into a fresh install of mysql:

% mysql -u [username] -p [database] < sqlDump.sql 
Enter password: 
ERROR 1153 (08S01) at line 1191: Got a packet bigger than 'max_allowed_packet' bytes

Which seemed related to blobs containing larger pdf files mostly (in my case at least). In trying to figure out how to get past this - seeing the default size for 'max_allowed_packet' was too low - in the daemon side (mysqld):

mysql> select @@max_allowed_packet;
+----------------------+
| @@max_allowed_packet |
+----------------------+
|              1048576 | 
+----------------------+

To fix this, I used the configuration files (optional) - located on my local system (OS X - Snow Leopard) at: /usr/local/mysql/support-files/my-xxx.cnf. I copied one of these (the 'my-small.cnf' specifically) to /etc/my.cnf, and edited the file to increase the default for the server to 64M:

# The MySQL server
[mysqld]
port            = 3306
socket          = /tmp/mysql.sock
skip-locking
key_buffer = 16K
max_allowed_packet = 64M
...

This will increase the limit (globally - since the file is located in /etc/my.cnf and not in ~/.my.cnf) for the server. This increased limit can then be seen here after a server restart (64*1024*1024):

mysql> select @@max_allowed_packet;
+----------------------+
| @@max_allowed_packet |
+----------------------+
|             67108864 | 
+----------------------+

After trying to re-import my file, I found out I also needed to have this via the client as well while importing, since setting the above limit did not seem to solve my issue:

%  mysql --max_allowed_packet=64M -u  [username] -p [database] < sqlDump.sql 

This finally worked in getting past the limitation I was hitting.

Maybe this will be a quick fix help to someone else running into this problem as well.

Blank lines in JSP output

I had posted some small entries on my old blog at Sun, and wanted to transfer some of those here for reference sake.


I was having an issue with jsp outputting blank lines at the top of output - and if the contentType being text/xml - causing parsing error being that the <?xml... directive not being the first line - causing the exception:

'XML or text declaration not at start of entity'

Come to find out, this had been addressed in JSP2.1 - but was a bit hard to track down.

Adding the line:

<%@page trimDirectiveWhitespaces="true"%>

to the top of your jsp will remove these, thus letting the XML feeds parse correctly.

Small fix - but has cured some headache in creating some feed proxies.

OSGi, JavaMail, and the mailcap issue

When developing some of the components for our application, I have been seeing some issues with ClassLoaders when creating them as OSGi bundles.

One main case that had me curious for a while was using JavaMail inside an OSGi bundle, and having to send a multipart mail.

The issue was - JavaMail relies on JAF (the activation framework), which houses a file (mailcap) in it's META-INF directory. So, if these (the javamail and jaf) are stored in separate bundles, then javamail cannot access the configuration file to determine which MIME types it can handle, and thus throwing an UnsupportedDataTypeException:

javax.activation.UnsupportedDataTypeException: no object DCH for MIME type multipart/alternative; 

An UnsupportedDataTypeException usually occurs because JAF cannot find the DataContentHandler (DCH) for a given MIME type by reading the mailcap.

Glassfish 3 actually bundles these together in one bundle (modules/mail.jar), but I was still having the issue described above.

So I went down the path trying to figure out what in the world I could do to get past this. You can't really export resources like you do packages in the manifest, so importing into my bnd file didn't work, and even trying to manually force new mailcaps (which seemed to work elsewhere) didn't work:

MailcapCommandMap mc = (MailcapCommandMap) CommandMap.getDefaultCommandMap();
mc.addMailcap("text/plain;; x-java-content-handler=com.sun.mail.handlers.text_plain");
mc.addMailcap("text/html;; x-java-content-handler=com.sun.mail.handlers.text_html");
mc.addMailcap("text/xml;; x-java-content-handler=com.sun.mail.handlers.text_xml");
mc.addMailcap("multipart/*;; x-java-content-handler=com.sun.mail.handlers.multipart_mixed; x-java-fallback-entry=true");
mc.addMailcap("message/rfc822;; x-java-content-handler=com.sun.mail.handlers.message_rfc822");
CommandMap.setDefaultCommandMap(mc);

This is basically just pushing through exactly what is in the mailcap file directly. But - this didn't work either. Odd...

I then went as far as create a new instance of the specific handler that is being used, and testing the support for that DataFlavor:

DataContentHandler dhmm = new com.sun.mail.handlers.multipart_mixed();
DataFlavor[] dtf = dhmm.getTransferDataFlavors();
for (DataFlavor tmpdf : dtf) {
 log.debug("   isSupported? " + tmpdf.getMimeType() + ":" + message.getDataHandler().isDataFlavorSupported(tmpdf));
}

And it shows it is supported: isSupported? multipart/mixed:true

Yet - when sending the message, same Exception. Ugh...

Finally, Sahoo (from the Glassfish team) gave me a suggestion of manipulating the ClassLoaders when I needed to to make the calls, saving the current ClassLoader so it can be put back into place.

In our bundle, we create the session and send the message in two different methods, so this had to be implemented twice, but finally - it worked!

// There is an issue in the OSGi framework preventing the MailCap
// from loading correctly. When getting the session here,
// temporarily set the ClassLoader to the loader inside the bundle
// that houses javax.mail. Reset at the end.
ClassLoader tcl = Thread.currentThread().getContextClassLoader();

try {
    // Set the ClassLoader to the javax.mail bundle loader.
    Thread.currentThread().setContextClassLoader(javax.mail.Session.class.getClassLoader());

    ...
} finally {
    // Reset the ClassLoader where it should be.
    Thread.currentThread().setContextClassLoader(tcl);
}

This is now working fine. I was a bit leery about mucking with the ClassLoaders in here - which was an issue with using JRuby code inside OSGi bundles as well, but this seems to be OK in that we are temporarily changing and immediately changing back.

The Pollers - MDB, Singleton, Glassfish, JRuby

It has been a bit since last posting - has been a whirlwind since then. :)

Ended up utilizing MDBs to be able to get past the issue with the way the app was using the pollers to listen to certain events in the application (observer) and processing them.

Keeping the original logic in ruby, I ended up using a Singleton Bean to create and store a rails instance that was shared with the same VM as the rest of the app. Also slated this Singleton to be instantiated at Startup (@Startup) so it would be available when the rest of the app was ready:

@Singleton
@Startup
public class EIScriptingContainer {


With this, created a separate MDBs for each ruby poller, giving commands via the message selectors to determine which poller to utilize. This gave us the ability to use a central "SystemManager" to send messages to a Topic that the MDBs were listening to, and depending on the serviceName, would know what to do:

@MessageDriven(mappedName = "jms/SysMgrReq", activationConfig = {
    @ActivationConfigProperty(propertyName = "messageSelector", propertyValue = "serviceName='EventNotifier' AND messageAction IS NOT NULL"),
    @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic")
})

I then bundled all of these up in a single jar and deployed as an app to GF.

A big issue was the fact that sometimes we needed to have more than one instance of a poller running, but when re-loading the original script (via the JRuby ScriptingContainer), all of the Rails stack was loading. That was where the Singleton instance of the ScriptingContainer was created and pre-loaded with the rails environment, and this was utilized each time in the stopping, starting, starting additional instance, and refreshing these pollers.

JRuby, JMS and Pollers

A common way in Ruby to utilize messaging is to use ActiveMessaging, and for ActiveMessaging to listen to queues/topics, etc. - implement "pollers", which will run as separate processes and listen/react to the messages.

ActiveMessaging takes these processors, and runs them in the environment (which is started externally) - creating new processes.

In trying to get everything in one box (VM), we needed a way to get around this, on top of using ActiveMessaging as a whole (since most of the app will be converted to java eventually).

In doing this, we took the poller processors and turned them into MessageListeners:

class QueueProcessor  < ApplicationProcessor
  include javax.jms.MessageListener

defined them as classes, and made sure they could run on their own and still function much like they did before:

class QueueProcessor  < ApplicationProcessor
  include javax.jms.MessageListener

  def initialize
    ...
  end

  def run
    # Instantiate a Sun Message Queue ConnectionFactory
    queueConnFactory = ConnectionFactory.new

    # Create a connection to the Sun Message Queue Message service
    queueConn = queueConnFactory.createConnection()

    # Create a session within the connection 
    queueSess = queueConn.createSession(false, Session::AUTO_ACKNOWLEDGE)

    # Instantiate a System Message Queue Destination
    # ToDo: Need to lookup via JNDI or some sort of aliasing.
    queueQueue = Queue.new("MyQueue")

    # Create a message consumer and listener
    queueMsgConsumer = queueSess.createConsumer(queueQueue)
    queueMsgConsumer.setMessageListener(self);

    # Start the Connection
    queueConn.start()
  end

  def onMessage(message)
    begin
      # Process our message as needed
      ...
    rescue
      @queue_logger.error "QueueProcessor caught #{$!} \n #{$!.backtrace.join("\n")}"
      ...
    end
  end
end

### Make sure to only run this process, and not any others.
if __FILE__ == $0
 queueProc = QueueProcessor.new
 queueProc.run
end

The next step in getting this to run separately, yet still in the same VM? I am thinking OSGi and ScriptEngine - so onto the next step to see if that is possible. :)

JRuby, JMS, OpenMQ, and Serialization

In working towards getting the current application running entirely within the Glassfish context, one of the issues I had to do was get the current implementation of the communication to the message queues away from ActiveMessaging/ActiveMQ into using JMS/OpenMQ.

There are a lot of fingers that ActiveMessaging has in here, so we will have to do some cleanup, but something that was causing me a few issues we figured out today was with the way the serializing of objects and placing in the queue as TextMessages was getting a bit out of whack when we converted some of the publishing models of ActiveMessaging to the sending of the message via JMS.

The serialization (which we are using Marshal dump/load for the ruby objects) was becoming an instance of TextMessageImpl (Java::ComSunMessagingJmqJmsclient::TextMessageImpl).

Simply pulling the text out of this on the jruby side fixed for us:
message = deserialize(message.getText())
and the deserialization worked ok.