Get more answers, solve problems, improve performance – faster – with Lumberjack
Finance and IT teams rely on system stability and strive for efficiency, especially during the financial close process. However, when trying to meet reporting deadlines, finance and IT teams often face obstacles, including system slow-downs or crashes. Finance would prefer to focus on running their reports, delivering them on time, and analyzing financial data. IT would prefer not to deal with system errors, crashes, or downtime.

Introducing Lumberjack, Finit’s easy-to-use, off-the-shelf, inexpensive EPM performance monitoring solution
Finit's infrastructure group has worked with clients to implement some of the most complex Oracle EPM environments in the United States. Through their many years of Oracle EPM experience, the Finit team identified obstacles and problem areas that are common to many companies. This was the catalyst for the team to develop a stand-alone, off-the-shelf tool that assists Oracle EPM users with system performance monitoring. From system tracking, system crash analysis, and log filtering, to infrastructure best-practices, analyzing degradation in performance and system errors, and managing alerts, this solution helps minimize system downtime while increasing efficiency, especially during critical financial close cycles.

What follows is an overview of Finit’s EPM performance monitoring solution, called Lumberjack, and offers EPM teams better visibility into their EPM environment and insights on how to minimize downtime and maximize performance.

What is Lumberjack
Lumberjack is a standalone application designed to benefit on-premise Oracle EPM Finance and IT teams by automating performance monitoring in order to help identify reasons why a system crashes, is slow, or is unstable. Lumberjack tracks and stores system activities and gives IT teams streamlined views into their logs, systems, and servers to more clearly see and quickly find where problems lie. In many cases EPM system monitoring and analysis can help teams prevent problems from happening. Lumberjack is quick and easy to set up, user-friendly, and does not increase burden or risk on resources.

The state of the EPM environment
Oracle EPM systems are time consuming to install and the EPM environment requires both Hyperion experience and IT involvement for proper monitoring and support. The complexity of Oracle EPM systems and the financial reporting environments present users with ongoing challenges. Multiple products and data sources can be difficult to manage and may introduce the potential for errors. Troubleshooting these errors can be difficult due to the configuration of the platform, complexity of the system, quantity of data being accessed, and number of concurrent users.

Reboot vs system monitoring
When it comes to performance, usage of the EPM system is greatly reduced during non-close periods. Then, during the financial close cycle users stress the system with heavy system activity in an effort to complete all necessary financial reports that must be submitted on a strict schedule.

Crashes are common during this crunch time. The finance department doesn’t have access to servers, and IT doesn’t truly understand what is happening in the EPM environment. IT may get alerted that there are crashes or system lags, or things are not working well. The default solution is simply to reboot or restart the whole system. Doing so may allow the reports to be run for a current close cycle, but a system reboot clears everything and refreshes memory. This means evidence that would point to the root cause of a system error or crash is lost, leaving no way to see or understand what happened, no way to identify what the proper fix should be, and an inability to diagnose and correct the issue when similar challenges arise in the future.

Instead of losing important information with a reboot or when logs roll over, it is ideal that performance activities are constantly captured. Performance monitoring allows EPM teams to identify what was happening when a problem arose and to access information that can help prevent problems from recurring. System performance monitoring offers a more consistently stable and reliable way to guard against ongoing issues.

Several system performance monitoring tools exist, but most are difficult to install, do not offer automated monitoring, and do not allow for efficient troubleshooting. There are utilities for monitoring servers to grab log files, but this still requires the labor-intensive work of sifting through the files. There are also tools that are EPM-specific rather than offering more general server monitoring. And, there are many solutions that are not specific to EPM. Most are time consuming to purchase, install, and set up – and, they are expensive.

The evolution of Lumberjack
Finit’s Lumberjack solution was initially developed to support the Finit infrastructure team in response to common and consistent issues their clients faced. It quickly became evident that many companies beyond our immediate clients could benefit from a nimble performance monitoring solution.

The team saw a need for a light service tool with optimal impact. A primary goal when developing Lumberjack was to develop a solution that is quick and easy to set up. In addition, they did not want to add time to the already tedious financial reporting and financial close process when solving for system monitoring. They wanted a solution that would not add on to each server, add additional stress to the environment, or add any additional processes for the finance and IT departments. In an effort to not reinvent the wheel, they aimed to create a solution that was lightweight and designed to rely on what was already built into the operating system, to piggy-back off Windows.

Lumberjack set-up and configuration is quick and easy
Lumberjack can be set up to monitor one or as many server environments as needed, including but not limited to production, development, QA, or any other on-premise EPM environment you may have. Once the list of servers is added and the monitoring tasks are selected [you tell it what to do], the set-up is complete. Configuration is relatively simple and takes approximately 15-30 minutes.

After Lumberjack is set up, data is collected from all logs and performance information is sent to Finit’s data center server. Finit then provides support w/ minimal impact on customer’s server or on their internal finance or IT team.

• Consolidations run slow
• Sudden degradation of performance
• There is a desire to optimize further even though performance may acceptable
• There is a need to analyze performance to find ways to improve efficiency
• There is a need to give access to CPU performance and RAM – across the total system and for individual EPM processes
• The need to more efficiently analyze the data in the log files

Log files
One of the key issues that EPM teams face when troubleshooting a crash or system error is log files. In an EPM environment, there are many files located on various servers. In fact, it is often not easy to determine where they live, or to determine which are important or which are even used. As infrastructure experts know, most log files are simply informational. Within each of the many log files there are tens of thousands of lines of data, much of this data is not pertinent to troubleshooting system performance. So, reviewing log files manually [is] [would be] intensely time-consuming, tedious, and would likely not result in an outcome that resolved the problem.

Lumberjack is configured to call out only the error conditions that will potentially cause problems. Lumberjack takes the relevant log files and filters the pertinent data, then pushes it into a central location. From there, reports in an excel-like pivot table format can be pulled. These reports contain both high level and raw, detailed information. Any one of the columns can be sorted using Lumberjack’s filtering features. This allows users to drill down to specific components, such as date and time to see only error messages for that period across all servers without requiring the user to know the exact server, folder, or location where that data resides. This format and functionality makes it easy to click on an entry and quickly drill down to identify the root cause of a problem. It also allows users to search all logs across all servers and look for specific error codes. This is especially beneficial when you know the common EPM errors that occur, such as the “spinning wheel of death”. Lumberjack stores error logs, so in the event that a system reboot is required, those logs can be retrieved to see where the error occurred and gives you access to information to determine why it occurred in order to prevent it from happening later.

• Ability to identify available RAM or changes in available RAM
• Ability to identify financial reports that are used
• Ability to easily see all EPM patches that are installed – and identify if there a server is missing a patch – with results displayed in English instead of a patch ID that requires translation

Lumberjack monitoring tools:  Lumberjack can show what standard performance monitoring tools show.

What a well-maintained, nicely performing environment looks like across several weeks 

Normal performance during close cycle


Normal system memory versus HFM app process         



HFM consolidation view - parsed log files  show server/POV details associated w/ any consolidation
(Can be captured w any performance monitoring tool.)

Lumberjack performance monitoring allows you to drill down to see much more.
What follows are some examples of what can be viewed within the performance reports in Lumberjack. 

Lumberjack view of pivot table format 



And, filtering data for “entity type and period”



Lumberjack drill down shows:
     Where slow-down occurred.
     Performance at certain times.



When consolidation was kicked off, the CPU went immediately to 100% utilization.



This chart in Lumberjack shows the amount of memory available on a server and flags a potential issue.
When consolidation started there was only 470MB memory on a server that has 16G of RAM.
While consolidation was running it shows the Windows server was trying to move memory in order to handle what HFM requires.
This shows that the server is trying to manage memory instead of doing task it was prompted to do.




This chart shows comparison of a consolidation run at different times.

• Our system is currently down. No one can connect to HFM, the “spinning wheel of death”, and some users can’t even log in.
• Our system has been unstable recently. Why?
• Where is the issue that is causing the system to lag?
• Our system became unusable and no one can log in. No one in the system could open forms or the HFM application. Can I look at log messages and identify an abnormality in environment?
• Why did a stuck thread message happen?
• Where do I go to resolve an issue?
• Is this a database issue or a development environment issue?

• Timeframe of issue occurrence and targets root of the issue allowing team to drill down further
• Other HFM database analysis
• Task audit, data audit, and errors
• Web logic server details
• Crash analysis
• Use log and performance data to enable team to do analysis after a crash

• Alerts - While monitoring the EPM environment LJ can be configured to send out alerts when certain issues or criteria are met (e.g. memory, disk space, etc.).

• Be used to determine if a system can handle the expected number of concurrent users, or how a system will respond to added users.
• Record actions and simulate stress on system to determine the potential impact of stress on a system.
• Predict go-live performance reality.

• Lumberjack exposes HFM programmer interface (API) in a different way.
• An added layer allows scripts built for 11.1.2.3 to be reused without modification in 11.1.2.4. Potential for use with Oracle’s cloud solution is also being explored.
• FDM loads to simulate processing concurrent end-users loading multiple files.
• HFM – multiple solutions including executing calculations, consolidations, and on-demand rules; process management; data file extract and load; execute extended analytics exports;
• LCM script commands to run LCM exports.
• Automate EPM patch installation – takes time consuming process from hours to minutes.

• Listing of logins to system cubes in your environment to see how active or inactive a cube is
• Display Smart View retrieves across all cubes – who and when accessed
• Display list of all calculations – makes it easy to find slow running calculations or those that degraded over time.

Summary

Get more answers – faster.
A slow EPM system and server errors waste valuable time and prevent you from meeting critical financial close deadlines. Introducing a tool that gives you visibility into your system, applications, and EPM logs so you can quickly solve problems when they happen, and even identify ways to prevent them from happening in the future.

Solve problems – faster.
Finit’s off-the-shelf EPM monitoring solution for companies using Oracle’s EPM offers an easy-to-use, inexpensive, and quick way to understand why a system crashed, to prevent common problems, to automate analysis, and to make troubleshooting efficient. This solution does not require additional processes on EPM servers, does not slow servers down, and is cost effective for the short and long term.

Improve performance – faster
When your goal is meeting deadlines for financial close reports, the last thing you need is a slow EPM system, or worse, a system crash. With Finit’s EPM System Performance Monitoring Solution, instead of lost information during server reboots, you have a tool that delivers valuable insight into your financial system.