Set Analysis Part 5b: Custom Groups in WebIntelligence

So how we do do this exactly? How can I use BusinessObjects to create custom groups on-the-fly? How can I get beyond my requirements for multi-pass SQL and speed up my reports? The answer is Sets.

Set Data

All the information regarding sets is located is the tables called sets_*. The most important tables are:

• set_set - contains the set name, description, etc.
• set_set_detail - contains all the set member information including member key, join date, leave date.

Here is a universe I created that you can download and use with the sample database provided in my earlier blog post. We need to add the set_set and set_set_detail tables to our WebIntelligence Universe so that the set member information they contain can be used within our query.

Here you can see that we created a join to the DW_CRM_CONTACT table. We also created a self-join on the set_set_detail table. The self join make sure that we reference the edate value of 12/31/2999, which is the default edate for current members of any set. In the case of a dynamic set, If a member had left the set then the edate would not be equal to 12/31/2999.

In addition to this, we also joined the set_set table to the set_set_detail table using the setversion_id key. (It may appear that the setversion_id is always the same as the set_id, but it is not so only join the tables via the setversion_id)

Here is a look at the Custom Product Groups

BEST PRACTICE: Whenever you may need to alias a table. If you alias a table, then all references to that table should be aliased. You can see that because I needed to reference the set_set_detail and set_set tables twice, once for products and once for contacts, I aliased both.

User Prompts

After joining the tables, we needed to create the filters for our custom groupings to make it easy for users to select which groups they would like to use on the report.

I specifically defined the filters a multi, which will allow the user to select more than one filter. This is important because it provides even more power for your end-users. Now they can do direct comparisons of one group to another, even if a member (e.g. contact) exists in more than one group. The only downside is that it means that a member can be counted twice, this is why I recommended that when you define your sets you should use naming conventions that identify the sets as being mutually exclusive.

Here is what the prompt will look like in action:

In my universe I did not use cascading prompts to first limit the sets by folder, but that is certainly an option since folder information is located in the set_project table.

Getting Results

After configurating the universe and setting up the prompts this is an example of the results you will see. I have run a report and selected multiple custom groups which are mutually exclusive. These groups on based on the current periods sales amounts:

Here you can see the report is broken down by my custom groups which were defined and processed by the Sets Engine.

Summary

I’ve really enjoyed taking you on a quick tour of Set Analysis and showing you some of the ways that you might get more value out of your existing data assets. Sets offers so many solutions around data analysis:

• Custom Groupings – which allows the organization to more nimble as changes such as mgmt structures, mergers and acquisitions take place.
• Classic Segmentation – Joiners, Leavers, Stayers (which in turn are perfect for analysis by a predictive tool so that you can see why are people joining or leaving? What are the influencers?)
• Advanced Analysis – “Give me all quad-pack customers who have never purchased phone accessories from a company owned retail store.”

The ability to divide customers up into categories of: highly profitable down through :unprofitable, and using that to drive the business because the bottom line of any business is the customer and your organization’s ability to make a profit.

Although dashboards are currently in vogue and everyone wants one, I often wonder how organizations are analyzing the information beyond the dashboards. I have nothing against dashboards, but I would encourage you to ask yourself this. Once we see that sales are falling, what am we going to do? How am we going to do root cause analysis? Were all customers affected? What about products? Segmentation might well hold the answer to these yet unsolved mysteries.

Set Analysis Part 5a: Custom Groups in WebIntelligence

I recently had a customer who was in the process of migrating from a legacy version of Microstrategy to BusinessObjects. They had been able to migrate over most of their core reports, but there were several reports that required a number of custom groups.

Microstrategy’s approach was to allow the customer to define any number of groups and filters from with the Microstrategy Desktop tool. These global filters and group definitions can then be leveraged by the report editor for building the report. Microstrategy’s SQL generation engine uses multi-pass SQL when creating the reports. Multi-pass SQL means that a single SQL statement cannot deliver back the correct results, so the report data is written into temporary tables. These temporary tables are then combined with additional queries to obtain the final results.

In the case of BusinessObjects, it wasn’t going to be possible to take the same approach. BusinessObjects does not generate multi-pass SQL. Instead, BusinessObjects focuses on creating extremely complex SQL in a single SQL statement. In most cases, this works just as well, but this was an exception. This sales report used a combination of multiple filters at multiple levels in the heirarchy. The Heirarchy was also defined within the custom groupings and not within the database.

The right answer is: Modify the Data Warehouse and ETL processes to make the underlying data better fit your reporting requirements. You see, the problem was that these custom groups should be defined within the data warehouse natively. This would allow for simplified SQL that can be run in a single pass. Translation? Reports the use custom groups will run a heck of a lot faster.

In my situation, the customer wasn’t going to be able to make these changes, but still wanted to migrate the report to BusinessObjects.

The best answer is: Use Sets. Sets gives you the ability to generate those custom grouping within the data warehouse. Unlike Microstrategy which will run the same multi-pass SQL over and over again, sets will allow us to “cache” the previous passes using sets and run single-pass SQL. We can then tie the set tables into the data warehouse and add the custom groupings to our universe.

The advantage of sets over ETL is that it can be much more dynamic. Sets can allow an analyst to create a new set in a matter of minutes and then see a report broken down by the new custom group.

In my case it was just what the doctor ordered. By introducing sets and adding the set tables to the BusinessObjects semantic layer, we now have access to the sets, which in this case, the customer called custom groups. It was perfect.

Now, not only will the customer be able to migrated this complex report from Microstrategy to BusinessObjects, but they will also be able to run the report much faster than ever before because the report will be generated through a single-pass process.

Tomorrow, we’ll take a look at how to set this up using our simplified data.

APOS: Providing the Missing Piece

When I was growing up I always enjoyed putting puzzles together. I loved the challenge of putting all those pieces together so that I could see the whole picture. Sometimes I would borrow a puzzle from a friend and as I was close to finishing the puzzle, I would discover 1 or 2 puzzle pieces were missing. The puzzle was incomplete.

From a BI perspective, APOS has been a long partner of BusinessObjects. My first opportunity to work with APOS came while I was working at Crystal Decisions. Our next generation Crystal Enterprise product was new and although it provided a great framework for the future, there were a few pieces missing from our complete BI picture. Thankfully, APOS developed a number of powerful solutions to fill those gaps so that our customers didn’t have to wait until tomorrow’s release for the functionality they needed today.

Do you feel you are missing a piece from BusinessObjects XI 3.0?

Business Objects has many great partners, but probably non-better than APOS. In January of this year, APOS was awarded the Business Objects Technology Partner of the Year Award for 2007. They have used the BusinessObjects SDK more extensively than any other BusinessObjects partner. They don’t just one solution. Their tools provide dozens of potential missing pieces that might fit your BI puzzle.

My Favorite APOS Tools

So what’s so great about APOS? Well, here are my top 3 list favorite solutions:

1. InfoScheduler - mass scheduling - dead simple. Scheduling has always been one of those areas were BusinessObjects added a few new features with ever release. Unfortunately however it’s not an area that tends to get a lot of attention. It was only in XI 3.0 that we introduced native bursting within the platform. InfoScheduler provides an Excel-based report scheduling tool that allows mass scheduling of reports from an Excel spreadsheet. How easy is that? Can you say zero training? Users can copy paste, import, export data into Excel to provide scheduling for 10, 100 or 1,000s of reports.
2. Object Manager - Although the CMC is a comprehensive administration console, administrators of large reporting environment often struggle with managing this volume of information. In XI 3.0, BusinessObjects introduced a number of new enhancements which made working with multiple objects much easier; however Object Manager takes it to the next level. You can manage users, groups, reports, security and even the report life cycle process. Object Manager provides power, ease-of-use and flexibility.
3. KPI - One of the most common requests I hear from customers is the ability to do reporting off their CMS. BusinessObjects XI provides an auditing database to allow customers to report off of user activity, but how do administrators report off the objects within the CMS? With KPI, APOS allows you to extract all the XML metadata that is stored within the CMS to an external relational database. Now you can create reports that allow you to:

• Analyze Business View and Universe configuration information
• Examine Crystal and Desktop Intelligence report structure
• See How many reports use the database field profit
• Show CMS Statistics
• Discover how many reports are scheduled to run tomorrow.

You can even set an automatic schedule to refresh the external metadata database on a regular basis. It’s your own mini-datamart for reporting off your BI system. You have to see it to believe it.

Southwest is using all three of these tools and several more!

Can’t Find Your Missing Piece?

If you’ve look through the APOS website and you can’t find the type of functionality that you feel is missing from BusinessObjects, call APOS anyway. Necessity is the mother of invention. Each of the solutions that APOS provides today was developed from the real-world needs of customers just like you. APOS would love to talk to you about extending BusinessObjects in a way that they may not have thought of.

Puzzle Complete

As a kid I was rarely able to find the missing puzzle piece, but in the area of extending BusinessObjects, APOS continues to deliver. Their depth of experience means they understand how BusinessObjects works today and how to make it work better for you tomorrow.

Why use a Semantic Layer?

Recently I had a customer ask me about using a semantic layer.

They said:

I am unaware of anyone using a metalayer (Universe) for Business Objects Enterprise XI. Most developers will build the report connecting directly to Teradata (or other sources; Oracle, SQL Server) and then schedule the report on XIR2. I will normally build the SQL first and then plug it into a Command Statement in Crystal Reports using dynamic date ranges to eliminate user selectable parameters for dates. Others do use parameters, but it’s discouraged due to both the response time from Teradata and processing overhead can not be managed as effectively.

I thought this was a great question. Sometimes you are get so comfortable with the technology that you think everyone recognizes the elements of a good Business Intelligence system and understands the advantages and disadvantages of different infrastructure elements. The semantic layer is definitely secret sauce of BusinessObjects and it is the primary reason why it was able to grow to become the largest provider of Business Intelligence software.

What is the Semantic Layer?

The semantic layer is a business translation layer that sits between the database and the end user. This means that user can interact with their data using familiar business terminology instead of having to understand where the data resides and what business rules to apply to the data.

What are the Advantages of a Semantic Layer?

Having a semantic layer on your database allows IT organizations to:

• Guarantees Correct Results - by applying rules to define database complexity and ambiguity. These rules drive the generation of the SQL and guarantee that if two users ask for the same information, they will get the same results.
• Guarantee Database Performance - by always generating the best SQL possible
• Guarantee User Understanding and Acceptance - by allowing users to understand how modifying their query will result in different results, while at the same time giving them independence from IT. The #1 complaint from most business organizations is the amount of time it takes IT to build reports for them. They want the independence to be able to build their own reports and know that the results will be correct.

A semantic layer will be able to create sophisticated SQL and in many instances may need to generate multiple SQL statements in order to return the correct results (chasm trap/fan trap). The semantic layer must understand how to deal with database loops, complex objects, complex sets (union, intersect, minus), aggregate table navigation and shortcut joins.

For more information about some of these concepts, there is an excellent presentation by Integra Solutions here

Most importantly, the semantic layer allows an organization to define a single version of the truth. That means that regardless of what application or user pulls information from the database, as long as they use the semantic layer, they will always get the same answer. Now that we are in a post-Enron/WorldCom age of corporate governance and Sarbanes-Oxley, it is more important than ever to make sure that there are clear definitions for revenue, margin %, churn rate, turnover, etc.

Implementing a semantic layer will put an end to conference pickering over who’s numbers are correct because now everyone can make sure they are using the same numbers.

What are the Disadvantages of a Semantic Layer?

Now, as much as I love all the business value that comes with the semantic layer, there is a cost. Prior to working for Business Objects, I was with Crystal Decisions and we didn’t have a semantic layer, so we had to figure out all the reasons why not the use it. (We did eventually release a semantic layer called Business Views in Crystal Enterprise 10). Therefore, I thought it would be good to share with you what we felt were the problems with a semantic layer.

The main contention was that the semantic layer was an “extra layer”. It’s not free. It has to be created, maintained and managed. It must be kept in sync with any database changes that occur. The other point was that you can use database constructs such as database views or stored procedures to accomplish much of what the semantic layer provides.

The final disadvantage to the semantic layer or at least the semantic layer of BusinessObjects was that it could only connect to one database at a time. In other words, if you had data in MS SQL Server and Oracle, you needed a semantic layer for both.

Semantic Layer Rebuttal

Although all these points were true, in fact the arguments were quite week. Although the semantic layer does have to be created, maintained and managed, it’s a lot easier to maintain 2 or 3 semantic layer definitions than it is to maintain 1,000’s of reports.

It also means that you don’t have to engage the database administrator directly to create views and stored procedures. Most large organizations would not be nimble enough to make changes quickly to constructs that live within the database. Typically these would not be able to be added or modified in a production database unless the next “major” database application release. In addition, you still have the issue of chasm traps and the need to generate multiple SQL statements if data exists in multiple fact tables. This type of complexity could not be handled by a database view alone and stored procedures are very inflexible. They must be programmed. The bottom line is that the SQL generated by a stored procedure or view is much more likely to be less efficient than the semantic layer.

It is true that BusinessObjects can only connect to one data source per semantic layer, but our report tools provide two alternatives you can either:

1. C reate a report pulling data from two universes and merge it in the report
2. Implement Data Federator which will pull information from two different data sources in real time using a single semantic layer definition.

Finally, a good semantic layer allows you to add query governors onto your queries so you can limit:

• how much data is pulled back
• how long a query will run
• who can connect to which dataset
• what row and column level security might be applied (e.g. I can only see “NY” data because I’m a manager in NY, or I cannot see the salary column in the HR database because I’m not a supervisor.)

Moving Beyond Relational Data

What about non-relational data sources like cubes? Well, cubes have their own built in metadata. Hyperion Essbase calls it an “Outline”. Its a definition of how the cube is constructed. The person who designs a cube must define what the dimensions and facts of the cube are going to be. Because the predescribed definition is required for a cube, the semantic layer in essence already exists in a cube.

In the case of BusinessObjects, we read in the metadata from the cube and build the equivalent structure within our semantic layer. In addition, we will change the type of data querying we perform. Instead of generating SQL for example, we will generate MDX for MS SQL Server and SAP BW.

Conclusion

A semantic layer is a key component is any truly successful Business Intelligence implementation. It will help you to deliver a single version of the truth to your business users, while providing the safety and security to your IT department to guarantee that users will only access data they are allowed to access, while now allowing them to accidently run a table scan against a 100 million row fact table. One of the hardest things about creating reports is knowing where to get the data and validating that the data is correct. The semantic layer delivers both of these in spades. Report developers can let the semantic layer ‘find’ the data for them and because it’s coming from the semantic layer, it must be correct because it’s already been validated by IT.

Remember: The semantic layer as the semantic center between your two halves of your BI cookie.

Hmmm. Good.