Thursday, October 14, 2010

How to use EJB 3 timer in a weblogic 10 cluster environment

Although there are some articles talking about EJB 3 timers or job schedulers , we were not able to find any detailed instructions on how to make EJB 3 timer work in a weblogic cluster. In this article we will go through a sample project to show how EJB 3 timers are used in a weblogic cluster. The sample project will create two recurring timers, the first recurring timer will periodically print out some simple information, the second recurring timer will create a couple of one-timer timers, each of which will print out some information. In this article, we will show you how to use weblogic admin console to configure the cluster, how the application uses the related configuration from the console and how to invoke timers, also explain what problems we faced and how we solved them.

Environment:   
web logic server 10.3.2, oracle database 11gR1, Eclipse Helios

Code:
Timer1SessionBeanLocal: local interface for creating timer
@Local
public interface Timer1SessionBeanLocal {
      public void createTimer();
}
Timer1SessionBean: a recurring timer that prints out something
@Stateless
public class Timer1SessionBean implements Timer1SessionBeanLocal {

      @Resource
      TimerService timerService;

    public Timer1SessionBean() {
    }

    public void createTimer() {
            timerService.createTimer(
                        60000,
                        60000, null);
      }
   
      @Timeout
      public void timeout(Timer arg0) {
            System.out.println("recurring timer1 : " + new Date());
      }
}

Timer2SessionBean: a recurring timer that creates a bunch of one-time timers,also the number of one-time timers created is roughly based on the maximum allowed number of active timers minus the number of active timers at that time.

@Stateless
public class Timer2SessionBean implements Timer2SessionBeanLocal {
      @Resource
      TimerService timerService;

      @EJB
      Timer3SessionBeanLocal timer3Bean;

      public Timer2SessionBean() {
      }

      public void createTimer() {
            timerService.createTimer(120000, 300000, null);
      }

      @Timeout
      public void timeout(Timer arg0) {
            System.out.println("recurring timer2 : " + new Date());

            // used to control the total number of threads running in the app
            // use 10 as maximum in this example.
            int numberOfActiveTimers = timer3Bean.getCountOfActiveTimers();

            if (numberOfActiveTimers < 10) {
                  int toCreateNum = 10 - numberOfActiveTimers;
                 
                  for (int i = 0; i < toCreateNum; i++) {
                        Timer3Info info = new Timer3Info();
                        // set start delays to be 30,60,90... seconds
                        info.setDelay(30000 * (i + 1));                                                                      
                        timer3Bean.createTimer(info);
                  }
            }
            System.out.println("Exit timeout in timer2");
      }
}

Timer3SessionBean: one-time timer created by another timer, provides the number of active timers for this bean, and prints out something.

@Stateless
public class Timer3SessionBean implements Timer3SessionBeanLocal {

      @Resource
      TimerService timerService;

      public Timer3SessionBean() {
      }

      public void createTimer(Timer3Info timerInfo) {
            timerService.createTimer(timerInfo.getDelay(), null);
      }

      @Timeout
      public void timeout(Timer arg0) {
            System.out.println("one-time timer3 : " + new Date());
      }

      /**
       *
       * @return the number of active timers
       */
      public int getCountOfActiveTimers(){
            int retVal = 0;
            try {
                  //In rare occasions, could throw NullPointerException //because of a bug in weblogic
                  @SuppressWarnings("unchecked")
                  Collection<Timer> timersCol = timerService.getTimers();
                 
                  if (timersCol != null)
                        retVal = timersCol.size();
            } catch (Exception e) {
                  //if it failed, use the maximum (10 in this example), so no //new timers can be created
                  retVal = 10;
            }

            return retVal;
           
      }
}


TestTimerCreateServlet: used to create recurring timers.

public class TestTimerCreateServlet extends HttpServlet {
      private static final long serialVersionUID = 1L;

      @EJB
      Timer1SessionBeanLocal timer1;

      @EJB
      Timer2SessionBeanLocal timer2;

      /**
       * @see HttpServlet#HttpServlet()
       */
      public TestTimerCreateServlet() {
            super();
      }

      /**
       * @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse
       *      response)
       */
      protected void doGet(HttpServletRequest request,
                  HttpServletResponse response) throws ServletException, IOException {
            System.out.println("start timer creation : " + new Date());
            try {
                  timer1.createTimer();
                  timer2.createTimer();
            } catch (Exception e) {
                  System.out.println("timer creation failed ");
                  throw new  RuntimeException("timer creation failed ", e);
            }
            System.out.println("Done timer creation : ");
      }

      /**
       * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse
       *      response)
       */
      protected void doPost(HttpServletRequest request,
                  HttpServletResponse response) throws ServletException, IOException {
            doGet(request, response);
      }
}

Overall, the code is quite simple. Some things are worth noting here is that local interfaces are used for the session beans, a servlet which needs to be invoked externally is used to create timers, also ‘timerService.getTimers()’ is used to find out the number of active timers for a session bean and also help control the number of running timers in the system, so the system will not be over stretched.

weblogic-ejb-jar.xml: some important configurations





 
<?xml version="1.0" encoding="UTF-8"?>
<wls:weblogic-ejb-jar xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/ejb-jar_3_0.xsd http://xmlns.oracle.com/weblogic/weblogic-ejb-jar http://xmlns.oracle.com/weblogic/weblogic-ejb-jar/1.0/weblogic-ejb-jar.xsd">
    <!--weblogic-version:10.3.2-->
    <wls:weblogic-enterprise-bean>
        <wls:ejb-name>Timer1SessionBean</wls:ejb-name>
        <wls:stateless-session-descriptor>
            <wls:timer-descriptor>
                <wls:persistent-store-logical-name>timerst</wls:persistent-store-logical-name>
            </wls:timer-descriptor>
        </wls:stateless-session-descriptor>
    </wls:weblogic-enterprise-bean>
    <wls:weblogic-enterprise-bean>
        <wls:ejb-name>Timer2SessionBean</wls:ejb-name>
        <wls:stateless-session-descriptor>
            <wls:timer-descriptor>
                <wls:persistent-store-logical-name>timerst</wls:persistent-store-logical-name>
            </wls:timer-descriptor>
        </wls:stateless-session-descriptor>
    </wls:weblogic-enterprise-bean>
    <wls:weblogic-enterprise-bean>
        <wls:ejb-name>Timer3SessionBean</wls:ejb-name>
        <wls:stateless-session-descriptor>
            <wls:timer-descriptor>
                <wls:persistent-store-logical-name>timerst</wls:persistent-store-logical-name>
            </wls:timer-descriptor>
        </wls:stateless-session-descriptor>
    </wls:weblogic-enterprise-bean>
    <wls:timer-implementation>Clustered</wls:timer-implementation>
</wls:weblogic-ejb-jar>

The most important things are that making sure the timers are cluster aware, and also using proper logical name for the persistent store  (timerst), which will be configured in the administrator console.


Admin console configuration:

Since EJB timers will be running in a clustered environment, weblogic uses two tables managing timers: ACTIVE and WEBLOGIC_TIMERS. These two tables can be named differently. These tables can be created automatically by the weblogic or can be created by you manually. Before you configure the cluster, you need to create JDBC data sources. 



Persistence of timers

There are two different styles of persistence. One is file based, the other is database based. We used the database for persistence. 



A table called ABC_WLSTORE will be created in schema ABCSYS automatically by the weblogic. It can also be created manually. 

The logical name (timerst) mentioned in the above screen shot is  (must be) exactly the same as the value of ‘persistent-store-logical-name’  in the weblogic-ejb-jar.xml. In our application we had two nodes in a cluster. The important lesson we learned was that creating only one persistence store that targets one migratable, as shown above, is the right thing to do. Do NOT create two persistent stores with each targeting one migratable. Do NOT create different persistent stores with the same data source and the same prefix name.


Deploy and Run

After deploying the application, you can invoke http://server:port/context/TestTimerCreateServlet to create timers, and then you can monitor the log files and/or the data in the WEBLOGIC_TIMERS table to find out how the execution went. 


Lessons learned

PostConstruct :  

We tried using a “PostConstruct” function in a session bean to create timers and had hoped that timers would be created when a session bean is deployed, but it failed. The sample code was
@PostConstruct
public void init() {
timerService.createTimer(60000,60000, null);
}

Exceptions:
Exception after create timer : java.lang.IllegalStateException: [EJB:010193]Illegal call to EJBContext method. The bean is in "ejbCreate" state. It cannot perform null action(s). Refer to the EJB specification for more details.
java.lang.IllegalStateException: [EJB:010193]Illegal call to EJBContext method. The bean is in "ejbCreate" state. It cannot perform null action(s). Refer to the EJB specification for more details.
at weblogic.ejb.container.internal.BaseEJBContext.checkAllowedMethod(BaseEJBContext.java:147)
at weblogic.ejb.container.internal.BaseEJBContext.checkAllowedToUseTimerService(BaseEJBContext.java:439)
at weblogic.ejb.container.internal.TimerServiceImpl.createTimer(TimerServiceImpl.java:82)
at weblogic.ejb.container.internal.TimerServiceImpl.createTimer(TimerServiceImpl.java:43)
at weblogic.ejb.container.deployer.TimerServiceProxyImpl.createTimer(TimerServiceProxyImpl.java:60)

Reason given by weblogic:

From the EJB 3.0 Specification, page 84, http://www.jcp.org/en/jsr/detail?id=220

The following steps describe the life cycle of a stateless session bean instance:

- A stateless session bean instance’s life starts when the container invokes the newInstance method on the session bean class to create a new session bean instance. Next, the container injections the bean’s SessionContext, if applicable, and performs any other dependency injection as specided by metadata annotations on the bean class or by the deployment descriptor. The container then calls the PostConstruct lifecycle callback interceptor methods for the bean, if any. The container can perform the instance creation at any time — there is no direct relationship to a client’s invocation of a business method or the create method.
- The session bean instance is now ready to be delegated a business method call from any client or a call from the container to the timeout callback method.
------

Therefore after the postConstruct callback has been executed the container can perform the instance creation at any time, so still it is in creating meanwhile the method createTimer doesn't allow that state.

Hence, the workaround that you mentioned at the beginning of calling the EJB initialization of Timer after the creation of the stateless bean is a better approach.

Finally the state of the bean that is triggering the IllegalStateException has correct behavior regarding to the EJB specification.


ServletContextListener

We tried using a ServletContextListener to create ejb timers during the context initialization.
@EJB  Timer1SessionBeanLocal timer1;
public void contextInitialized(ServletContextEvent arg0) {
                timer1.createTimer();
}
It failed also.

Exception:
javax.ejb.EJBException: EJB Exception: : java.lang.NullPointerException at weblogic.ejb.container.timer.ClusteredEJBTimerManager.createTimer(ClusteredEJBTimerManager.java:76) at weblogic.ejb.container.timer.ClusteredEJBTimerManager.createTimer(ClusteredEJBTimerManager.java:95)
at weblogic.ejb.container.internal.TimerServiceImpl.createTimer(TimerServiceImpl.java:125)
at weblogic.ejb.container.internal.TimerServiceImpl.createTimer(TimerServiceImpl.java:49)

Reasons:
The weblogic team asked us to apply a patch for this problem. But we never did because of some other constraints.


timerService.getTimers()

Sometimes this function call throws a NullPointerException

weblogic.ejb.container.deployer.TimerServiceProxyImpl@3a2229fc
java.lang.NullPointerException
        at weblogic.scheduler.ejb.internal.EJBTimerManagerImpl$TimerWrapper.getListener(EJBTimerManagerImpl.java:187)
        at weblogic.ejb.container.timer.ClusteredEJBTimerManager.getTimers(ClusteredEJBTimerManager.java:123)
        at weblogic.ejb.container.timer.ClusteredEJBTimerManager.getTimers(ClusteredEJBTimerManager.java:107)
        at weblogic.ejb.container.internal.TimerServiceImpl.getTimers(TimerServiceImpl.java:158)
        at weblogic.ejb.container.deployer.TimerServiceProxyImpl.getTimers(TimerServiceProxyImpl.java:74)

Reason:
     The weblogic support team did not give a good explanation. It could be a bug in the weblogic.

Server stop/restart

As long as there is one server in the cluster still running, the timers will still function,  the application may run slower though. When all servers in the cluster stop, the timers will stop running.  When at least one of the servers in the cluster restarts again, the timers will start running.

Application stop/restart

After the EJB 3 timer application stops, the timers will stop running. When the application restarts, the timers will NOT start running. The weblogic support team said it may be a bug.


Application undeployment/redeployment

When an EJB 3 timer application is undeployed, the timers will be removed from the system, the records related to the timers for this application in the WEBLOGIC_TIMERS table will be deleted. Sometimes, some records related to this application will still exist in the ACTIVE table (it may be a bug), but it won’t affect the future deployment of the application. When the application is redeployed, a new cycle starts again. We did not use “update” for the application redeployment often, we felt it should be avoided from our experience.
Overall, we think stopping and then restarting an EJB 3 timer application should be avoided.  Use the server stop/restart or application undeployment/redeployment.


EJB 3.1 Timer

There are nice articles online regarding this new technology. Overall, the syntax is richer, the timer can be created declaratively. Since we do not have much experience on this, we do not have any to share on this.


Conclusions

Overall, EJB 3 timer provides a nice way to schedule tasks in jee environment. How to create ejb timers and use them effectively in a clustered environment is not very straightforward. We went through a sample application,  explained the implementation and configuration in  weblogic 10.3.2, and also shared the lessons we learned. We hope this will provide some useful information to users when they use EJB 3 timers  in their applications.


Acknowledgements

Big thanks to my colleague Lee Slezak at HP for constructive suggestions.




23 comments:

  1. There may be minor bugs in the Weblogic server regarding EJB 3 timer implementations, but overall, it works very well. We have a big application running in the production, which uses a lot of EJB 3 timers. It has been running for a while without any glitches.

    ReplyDelete
  2. Great blog post, will definitely be making direct use of this very soon, thanks

    ReplyDelete
  3. Did you find a way to automatically create the timer after a successful application startup? Invoking a servlet manually is not an acceptable way.
    My first approach was also the @PostConstruct (which failed of course).
    My second approach was to do it in an ApplicationLifecycleListener#postStart, but at this time, the TimerService isn't injected already (it is null).

    ReplyDelete
    Replies
    1. For WebLogic you can create a postStart() listener in the ApplicationLifecycleListener. From here create the timer as required.

      Delete
  4. OK - the good old HttpServlet#init works. I call now the createTimer in the init method of a servlet, which has load-on-startup set to 1. Then the timer is initialization works fine.

    ReplyDelete
    Replies
    1. I am not sure whether init() will work as you expected. My understanding is: One container will create one timer. So if you have a cluster that has two containers, two identical timers will be created, even though you may just want to have one timer created in one CLUSTER.

      Delete
  5. Thanks information for very useful

    ReplyDelete
  6. Hi Guys,
    Did you got any annswer or explanation for below error:

    timerService.getTimers()

    Sometimes this function call throws a NullPointerException

    weblogic.ejb.container.deployer.TimerServiceProxyImpl@3a2229fc
    java.lang.NullPointerException

    Thanks in advance....

    ReplyDelete
    Replies
    1. Oracle was never able to provide any reasonable explanation about this. We never had a fix to this problem but managed to get around this in the application.

      Delete
  7. the jdbc must be configured to not support Global Transactions

    The JDBC data source or multi data source used by this JDBC store to access the store's database table (WLStore). This data source or multi data source must be targeted to the same server instance as the JDBC store.

    Note: You cannot specify a JDBC data source that is configured to support global transactions. Therefore, the specified JDBC data source must use a non-XA JDBC driver. You also cannot enable Logging Last Resource or Emulate Two-Phase Commit in the data source.

    ReplyDelete
    Replies
    1. I was not aware of this restriction. As far as I can remember, the application did not have any need to support global transactions.

      Delete
  8. How i Use this time implementation in Oracle Service bus, to call the external service in predefined intervals?

    Thanks ,
    Rajashekar

    ReplyDelete
    Replies
    1. I do not know how to do this in oracle service bus since I am not familiar with this product. Oracle service bus may have time-based functionality you can use directly instead of relying on EJB timers.

      Delete
  9. Hi,
    First of all, Thanks for the blog.
    I am searching for a way to deploy the EJB timers in a cluster with file based style of persistence.

    I have provided the name of correct file store in the weblogic-ejb-jar.xml. But
    it still goes and stores the timer objects in WEBLOGIC_TIMER table in the DB. (I have a datasource mentioned in the CLUSTER->Scheduling tab in weblogic console)
    Kindly point if I am doing any thing wrong or I am missing any thing.

    ReplyDelete
    Replies
    1. I am not sure whether the file based persistence is supported for a cluster. If it is supported, at least it means the file or files have to be shared across cluster instances. Even though technically it is doable, it may have some other challenges to overcome, for example, security, locking and so on.

      Delete
  10. hi Shaoxiong Yang,

    Do you have stepwise guide for deploying EJB 3.1 automatic singleton Timers (using @Schedule)on cluster environment?

    ReplyDelete
  11. Excelent example. I have a doubt by ACTIVE and WEBLOGIC_TIMERS tables. Do you need any extra settings for created automatically by the weblogic?. Sorry for my bad English

    ReplyDelete
    Replies
    1. Not sure what you meant. As far as I remember, there was no extra settings.

      Delete
  12. hi Shaoxiong,Thanks for your detailed article.We have an issue with timer on cluster.Looks like timer is running twice.can you pls guide me how to control timer to run only once

    ReplyDelete
    Replies
    1. I am not sure how to guide you on this since this has been a while for me. You may need to take a look at the relevant records in the tables mentioned in this article in your project. You can even manually remove them and start redeploying your application from scratch.

      Delete