For more than 50 years, Auerbach Publications has been printing cutting-edge books on all topics IT.

Read archived articles or become a new subscriber to IT Today, a free newsletter.

This free newsetter offers strategies and insight to managers and hackers alike. Become a new subscriber today.



Interested in submitting an article? Want to comment about an article?

Contact John Wyzalek editor of IT Performance Improvement.


Big Data Building Blocks to Decision Support

Stephan Kudyba, New Jersey Institute of Technology
Matthew Kwatinetz,QBL Partners

You may ask: Why are there classifications of data? Isn't data simply data? One of the reasons involves the activities required to manage and analyze the resources that are involved in generating value from it. Yes, big data sounds impressive and almost implies that value exists simply in storing it. The reality is, however, that unless data can help decision makers make better decisions, enhance strategic initiatives, help marketers more effectively communicate with consumers, enable healthcare providers to better allocate resources to enhance the treatment and outcomes of their patients, etc., there is little value to this resource, even if it is called big.

Data itself is a record of an event or a transaction:

  • A purchase of a product
  • A response to a marketing initiative
  • A text sent to another individual
  • A click on a link

In its crude form, data provides little value. However, if data is corrected for errors, aggregated, normalized, calculated, or categorized, its value grows dramatically. In other words, data are the building blocks to information, and information is a vital input to knowledge generation for decision makers (Davenport and Prusak, 2000). Taking this into consideration, the "big" part of big data can actually augment value significantly to those who use it correctly. Ultimately, when data is managed correctly, it provides a vital input for decision makers across industry sectors to make better decisions.

So why does big data imply a significant increase in the value of data? Because big data can provide more descriptive information as to why something has happened:

  • Why and who responded to my online marketing initiative?
  • What do people think of my product and potentially why?
  • What factors are affecting my performance metrics?
  • Why did my sales increase notably last month?
  • What led my patient treatment outcomes to improve?

Source of More Descriptive Variables

Big data implies not just more records/elements of data, but more data variables and new data variables that possibly describe reasons why actions occur. When performing analytics and constructing models that utilize data to describe processes, an inherent limitation is that the analyst simply doesn't have all the pertinent data that accounts for all the explanatory variance of that process. The resulting analytic report may be missing some very important information. If you're attempting to better understand where to locate your new retail outlet in a mall and you don't have detailed shopper traffic patterns, you may be missing some essential descriptive information that affects your decision. As a result, you locate your store in what seems to be a strategically appropriate space, but for some reason, the traffic for your business just isn't there. You may want to know what the market thinks of your new product idea, but unfortunately you were only able to obtain 1000 responses to your survey of your target population. The result is you make decisions with the limited data resources you have. However, if you text your question to 50,000 of your target population, your results may be more accurate, or let's say, more of an indication of market sentiment.

As technology continues to evolve and become a natural part of everyone's lives, so too does the generation of new data sources. The last few years have seen the explosion of mobile computing: the smartphone may be the most headlining example, but the trend extends down to your laundry machine, sprinkler system, and the label on the clothing that you bought retail. One of the most unexpected and highest impact trends in this regard is the ability to leverage data variables that describe activities/ processes. We all know that technology has provided faster, better computers— but now the trend is for technology to feed in the generation of never before seen data at a scale that is breathtaking. What follows are some brief examples of this.

The following illustrations depict the evolution of big data in various industry sectors and business scenarios. Just think of the new descriptive variables (data resources) that can be analyzed in these contemporary scenarios as opposed to the ancient times of the 1990s!

Industry Examples of Big Data


In some recent political campaigns, politicians began to mobilize the electorate in greater proportion than ever before. Previously, campaign managers had relied unduly on door-to-door recruiting, flyering in coffee shops, rallies, and telemarketing calls. Now campaigns can be managed completely on the Internet, using social network data and implied geographic locations to expand connectivity between the like-minded. The focus is not just on generating more votes, but has extended to the ever-important fund-raising initiatives as well. Campaigners are able to leverage the power of big data and focus on micro-donations and the viral power of the Internet to spread the word—more dollars were raised through this vehicle than had been seen in history. The key function of the use of the big data allowed local supporters to organize other local supporters, using social networking software and self-identified zip code and neighborhood locations. That turned data resources locational, adding a new dimension of information to be exploited, polled, and aggregated to help determine where bases of support were stronger/weaker. Where will it go next? It is likely that in the not-so-distant future we will find voter registrations tagged to mobile devices, and the ability to circumvent statistical sampling polls with actual polls of the population, sorted by geography, demography, and psychographics. Democratic campaign managers estimate that they collected 13 million email addresses in the 2008 campaign, communicating directly with about 20% of the total votes needed to win. Eric Schmidt (former CEO of Google) says that since 2008, the game has completely changed: "In 2008 most people didn't operate on [Facebook and Twitter]. The difference now is, first and foremost, the growth of Facebook, which is much, much more deeply penetrated . . . you can run political campaigns on the sum of those tools [Facebook, YouTube and Twitter]" (quotes from Bloomberg Business Week, June 1824, 2012; additional info from Tumulty, 2012).

Investment Diligence and Social Media

"Wall Street analysts are increasingly incorporating data from social media and Internet search trends into their investment strategies" ("What the Experts Say," 2012). The use of social media data is generally called unstructured data. Five years ago, surveys showed that approximately 2% of investment firms used such data—today "that number is closer to 50 percent" (Cha, 2012). The World Economic Forum has now classified this type of data as an economic asset, and this includes monitoring millions of tweets per day, scanning comments on buyer sites such as Amazon, processing job offerings on TheLadders or, etc. "Big data is fundamentally changing how we trade," said financial services consultant Adam Honore (adhonore, Utilizing the number and trending features of Twitter, Facebook, and other media platforms, these investors can test how "sticky" certain products, services, or ideas are in the country. From this information, they can make investment choices on one product vs. another—or on the general investor sentiment. This information does not replace existing investment diligence, but in fact adds to the depth and quality (or lack thereof sometimes!) of analysis.

Real Estate

Investment dollars in the capital markets are split between three main categories, as measured by value: bonds, stocks, and alternative assets, including real estate. Since bonds were traded, an informal network of brokers and market makers has been able to serve as gateways to information, given that many transactions go through centralized clearinghouses. In 1971, NASDAQ was the first stock market to go electronic, and as the information revolution continued, it soon allowed for any person around the world to sit at the hub of cutting-edge news, information, and share prices. After a particular tech-savvy Salomon Brothers trader left that company, he led the further digitization of data and constant updating of news to create a big data empire: Michael Bloomberg. Real estate, however, has been late to the game. To understand real estate prices in any given market has been more challenging, as many transactions are private, and different cities and counties can have significantly different reporting mechanisms and data structures. Through the late 1980s and 1990s, real estate was often tracked in boxes of files, mailed back and forth across the country. As cities began to go digital, a new opportunity was created. In the year 2000, Real Capital Analytics ( was founded by Robert White to utilize data mining techniques to aggregate data worldwide on real estate transactions, and make that data available digitally. Real estate research firms have many techniques to acquire data: programmatically scraping websites, taking feeds from property tax departments, polling brokerage firms, tracking news feeds, licensing and warehousing proprietary data, and more. All of these sources of data can be reviewed on an hourly basis, funneled through analysis, and then displayed in a user-friendly manner: charts, indices, and reports that are sorting hundreds of thousands of daily data points.

Specialized Real Estate: Building Energy Disclosure and Smart Meters

Over 40% of energy use and carbon emissions in the United States come from existing buildings ( index.cfm). To put this in perspective, if you combined the energy use and emissions output of all of the SUVs on the road in North America, this would be approximately 3%. So you can see that the use of energy by existing buildings is a very important piece of data. Until recently, this data has been held in many different databases for utilities across the country, with no central repository or easy means for reconciling these data sets. Today, three trends have picked up: (1) energy disclosure ordinances, (2) satellite heat map data, and (3) data warehousing aggregations based on smart meters. The amount of data needed here to control for effective information is staggering: any analysis must account for building size, use, geographic location, seasonality, climactic variation, occupancy, etc. In many of these cases, information is collected on a granularity of 115 minutes! That is for every building, in every city, in every state in the country: billions of data points per day (

Commerce and Loyalty Data

When you walk into your favorite retail outlet—be it clothing, jewelry, books, or food—there is nothing quite as gratifying as being recognized, your tastes being understood, and receiving personal service ("The usual, please!"). In the distant past, this probably meant a neighborhood shop where you literally were known by the salesperson. In the 1990s this was transformed into a "loyalty program" craze in which large-scale (franchised, national, international) retailers were able to tag your shopping to a digital ID card that they enticed you to use by offering discounts. But Internet commerce, under the thought leadership of Amazon, transformed this experience entirely. Once you are online, not only can a retailer track your purchases, but it can track what products you look at, things you plan to buy (wish lists), items you buy for others (registry), and even what pages you spend time on and for how long. This provides retailers with a competitive advantage: they can tailor your shopping experience and suggest new products. Witness Netflix recommendations, Pandora's preference algorithms, and LinkedIn's suggestion of who you might next want to connect with or apply for a job from. Moreover, it is not just information from their own site that these online merchants can now pull from— the trend has now reclaimed the point-of-sale data from brick-and-mortar stores as well. Retailers integrate physical data with online point-of-sale data, and can also view what other sites you visit, where else you make purchases, who makes purchases for you, and what "like-minded shoppers" may be in your network.

Crowd-Sourced Crime Fighting

In an effort to aid local policing efforts, policing has found a new ally: you! Over the last decade "hot spot" policing has become the effective leading strategy for reducing crime: take careful record of where crime occurs, measure density regions, and overwhelm the highest density regions with extremely quick and overpowering responses. However, this strategy still relies on actually being able to track all of the crime incidents—no small task, as the force's personnel have limited resources. Enter the crowd sourcing platforms. Some cities have created apps for mobile devices (or other interfaces) that allow individual citizens to upload information that indicates crimes they have witnessed ( augusta)! The upload contains the description of the crime, a geographic location, and a time stamp. As participation increases, so too do "eyes on the street," and the map is filled with the information needed to improve police performance.

Pedestrian Traffic Patterns in Retail

Thanks to some recent controversies, you probably already know that your cell phone allows you to be tracked at nearly any time of day, provided it is powered on. While privacy laws currently still protect you from being identified with this feature (without your opting in), new technologies are available to identify unique movements. Cell tower "repeaters" in strategic locations in malls and downtowns can track "unique cell phones" and their walking patterns. As a result, a mall owner might want to know how many people take the elevator vs. the stairs—and of the ones who take the elevator, do they ever walk by the store on the other side of it? Further, if they find a patron lingering in the leather goods section of the store for more than 12 minutes, but that customer does not stop at the cash register, they will send a text message advertisement promotion to the customer's phone before he or she leaves the store, offering a discount on—you guessed it—leather goods. This is only the beginning of this technology. Expect to see it deployed in cities to track crime patterns, the safety of certain intersections, and more (http:// retail-stores-by-pinging-peoples-phones/; ex-googlers-raise-5-8-million-to-help-retailers-track-foot-traffic/).

Intelligent Transport Application

New applications being developed for smartphones pool voluntarily offered information from unique sources into a real-time database providing an instant advantage from the use of big data. Uber, a mobile phone-based transportation application, connects drivers (of limousines, taxis) with potential passengers. As each driver "opts in" to uber from his or her phone, the phone sends a GPS signal update to the master Uber map. When a passenger is ready for a ride, the passenger turns on his or her Uber signal and effectively puts out an electronic thumb. Both passenger and driver receive an instant updated map with the potential matches to be found as moving dots across the map, with estimates of congestion (which influence pricing), as well as arrival information. In a similar fashion, Waze is a transport application for local drivers. When drivers get in their car, they turn on Waze, which utilizes the phone's GPS tracker, motion sensors, and built-in geographic road information (speed limits, lights, stop signs) to estimate the level of traffic you are experiencing while driving. Waze then merges your information with all other local drivers' information, creating a real-time picture of road traffic. The application also allows for the reporting of police presence, traffic, accidents, and not-to-miss sights! In essence, this application creates a virtual cloud of self-reported big data.


Cha, A.E. "Big Data" from Social Media, Elsewhere Online Redefines Trend-Watching. Washington Post, June 6, 2012.

Davenport, T., and Prusak, L.Working Knowledge. Harvard Business Review Press, Boston, Massachusetts, 2000.

Tumulty, K. Twitter Becomes a Key Real-Time Tool for Campaigns. Washington Post, April 26, 2012.

Read more IT Performance Improvement

This article is an excerpt from:

There is an ongoing data explosion transpiring that will make previous creations, collections, and storage of data look trivial. Big Data, Mining, and Analytics: Components of Strategic Decision Making ties together big data, data mining, and analytics to explain how readers can leverage them to extract valuable insights from their data. Facilitating a clear understanding of big data, it supplies authoritative insights from expert contributors into leveraging data resources, including big data, to improve decision making.

Illustrating basic approaches of business intelligence to the more complex methods of data and text mining, the book guides readers through the process of extracting valuable knowledge from the varieties of data currently being generated in the brick and mortar and internet environments. It considers the broad spectrum of analytics approaches for decision making, including dashboards, OLAP cubes, data mining, and text mining.