Thoughtput

Seeing the Forest and the Trees

A quick introduction to understanding, improving, and managing performance

Diagnosing and troubleshooting system performance across hundreds or thousands of application servers and dozens of NFS file servers is a big deal. Historically, it has been as much art as science because the visibility of I/O traffic across all the end points was not easily consolidated. Sure, network traffic analyzers exist, but digging in to the I/O activity is a whole different story.

The hardest part is that NAS devices present statistics on a per device basis. Application servers present statistics on a per server basis. Evaluating these end points does not provide an aggregated view. With many applications running in clustered or virtualized compute environments, and with data sets spanning multiple storage systems, it can be very difficult to get a complete picture.

Scalable caching appliances from Gear6 help customers see the forest as well as the trees. By placing a centralized resource in the network that can serve data from any storage system to any application server, the caching appliance is uniquely positioned to deliver aggregate data center statistics. This provides an invaluable resource to administrators trying to understand their overall system performance.

After understanding comes improvement. Here caching appliances enhance existing NAS systems by placing frequently accessed data in memory instead of disk, enabling much faster response times, and ultimately faster application performance. Finally, a set of management tools help customers monitor and adjust performance over time as workloads grow and change in makeup. These steps are outlined in the performance improvement cycle.

Often we hear the question, why can't I just put some more cache memory on my application server or my storage system?

You could. But then you are back to square one in terms of getting a true understanding of the entire system environment. Do you really want to monitor the cache utilization across hundreds of application servers and dozens of storage systems? Or might it be more effective to manage that in one place?

There will always be places to enhance performance across the food chain of applications, servers, caching, networks, and storage systems. But to fully assess and make use of a valuable resource like caching, the network is a smarter place to locate the resource. Not that you couldn't or shouldn't place caching resources on end-node devices, but rather that you should place a valued resource where it can scale and be shared across the maximum number of devices.

Please Don't Move My Data

Moving data can be a pain for end users, as well as data center and storage administrators. Specifically, any time a data set needs to be moved from one persistent storage device to another, there are numerous items requiring administrative attention. We spoke about not having to move data in an earlier post titled How About My Snapshots? Gear6 sees the ability to increase data performance without data movement as a key value for customers.

For those interested in the detail on data movement challenges, we've compiled the following reasons why data movement is possibly the #1 IT challenge today.

• Performance management
  • Many traditional performance improvement measures require moving data
    
• Cost management
  • Managing and maintaining multiple tiers (Tier-1 -> Tier-2 -> Tier-3) can reduce hardware costs, but is expensive to manage, particularly due to data identification and movement challenges
  • Lease returns require data movement and can be costly if equipment is not returned in time
  • Technology upgrades/refreshes require data movement, thereby adding to the cost of equipment acquisition
    

• Impact of moving data
  • IT time required to negotiate with internal users before moving data
    • It takes the same negotiation time for 100GB of data as it does for 1TB of data
    • Data is usually moved at the mount point level as opposed to the entire volume to mitigate user impact
    • Determining customers impact of each data move can be very tricky, particularly maintaining user and group level permissions consistently
  • Time lost for the IT administrator and all users during data migration job
    
    • Some pre-moves are possible but the final data synchronization can take hours to complete. Typically, no access to the data can occur at this time
  • Upon data move completion, a scan of the entire environment is required to remount the clients to use the new data location
    • This can be a time consuming task in environments with a large number of clients, and in some cases clients may need to be rebooted
  • Production network bandwidth and storage performance suffers with data moves
    • To reduce downtime, moves are typically done as fast as possible, consuming network bandwidth to the source and destination devices, as well within the core network. CPU levels are also impacted on the source and destination devices as well as networking equipment
  • IT administrators need to ensure that all IT infrastructure applications are aware of the planned data move
    
    • Backups are a great example. Once the data is moved, the new data needs to be backed up from that new location, and a history of the data move must be maintained to ensure recovery to a specific point in time
  • User/job impact time to move data goes beyond just the data movement portion and includes:
    • Time for customers to stop using the data + time to move data + time for customers to ramp back up for full production = Total Impact
      • Customers often move on to other things during the move impacting productivity
      • Example: 1) Customer ramps down on Friday around noon for a 7:00 PM data move due to job length; 2) Data move occurs at 7:00 PM; 3) Customer re-starts jobs on Monday morning at 7:00 AM = 67 hours of lost productivity for that one user. The data move may have only taken 2 hours
    • For multiple data moves users/jobs could be impacted multiple times over multiple days
• Data moves typically occur after hours to reduce direct user impact, but this still impacts lights-out operations during these periods. This has larger ramifications to IT teams
• Movement of data between two different vendors is difficult as all data-movement technologies are not 100% compatible. Therefore the time to move can be significantly longer

Whew! If you made it this far, congratulations! You are now prepared for the full suite of data movement challenges. If the list seems daunting, be sure consider alternatives to improving performance without having to move data. More info can be found at www.gear6.com.

Centralized storage; it’s a good thing!

Many companies have engineering teams working with data intensive applications that can’t get the performance they need via centralized storage architectures. In other words, they become disk I/O bound.

I spoke to a sharp systems manager the other day that supports the data storage for a large team. He knows first-hand how developers value performance, and finds they often squeeze performance by installing additional local drives in their workstations for fast storage. But this is hard to backup and manage centrally.

He also knows that right now he can’t provide the same level of performance from the centralized infrastructure – which is much easier and more cost-effective to manage and protect. So it is hard for him to force the development team to use centralized storage. In essence, he’d be asking them to sacrifice precious performance.

This is another interesting twist on storage performance, showing symptoms that ripple through organizations. The developer’s desk USED to be the best option for fast storage as centralized storage was too cumbersome. But large enterprises need storage solutions that effectively support data intensive development efforts AND provide IT and storage administrators with centralized management and protection.

About Reality Check

We added a new category today called Reality Check. The goal is to provide short snippets of wisdom from the real world. As I like to remind myself, you can learn more talking to an IT wiz for 5 minutes in person than you can from spending a week in the office researching on the Internet. Enjoy a slice of reality!