23: Performance Analysis with the New Relic RPM

Summary: <h2>Goals</h2> In this lesson, we take a look at techniques for monitoring and improving the performance of your Rails application. Please note that, while we’ve tried to make these notes complete, they aren’t the full tutorial; that’s in the screencast, which you can access via the button on the left. We recommend that you right-click on the button and choose Save As to download it to your computer. This lesson does not depend on the sample application code, although you can start with the sample application code (in the <a href="http://github.com/chaupt/learning-rails-sample-app">Github repository</a>) to explore the developer mode of the New Relic plugin, as we do in this screencast. <h2>Improving the Performance of Your Rails Application</h2> For most non-trivial Rails applications, there’s going to be some performance issues. Rails has been subject to a lot of performance criticism, but in reality, Rails can perform very well—it just takes some tuning to get there. Rails makes it quick and easy to get your application up and running, and in the process it also makes it easy to ignore a lot of issues that require some attention when performance is critical. Fortunately, there are plenty of tools and techniques at your disposal when it comes to performance tuning. You’ll need to look for slow queries, for queries that are being performed too frequently, and for opportunities to cache pages and parts of pages. In this screencast, which is sponsored by <a href="http://newrelic.com">New Relic</a>, we’re going to focus on the tools and service that New Relic provides. A word about our sponsorships: we think of this like public radio. Sponsors underwrite the cost of producing and distributing a lesson, and we give them opportunities to review the content. But the words and opinions are all ours, and we wouldn’t say anything we didn’t personally stand behind. New Relic is being used by a lot of folks who we’d trust to have made a good choice: <a href="http://37signals.com">37signals</a> (Basecamp and other apps), <a href="http://twitter.com">Twitter</a>, <a href="http://lighthouseapp.com">Lighthouse</a>, and <a href="http://shopify.com">Shopify</a>, to name a few. It’s also included in all hosting packages at <a href="http://engineyard.com">Engine Yard</a>. There are alternatives to New Relic’s service, such as those from <a href="http://fiveruns.com">Five Runs</a> and the recent entrant from <a href="http://rubysophic.com">Rubysophic</a>. There’s related services, such as Scout, and a variety of open source performance tuning tools, such as http-perf. For more information on a variety of performance optimization tools and techniques, see the <a href="/topic/24292-performance-tuning-for-ruby-on-rails">performance tuning</a> section of the BuildingWebApps site. <h2>Installing the New Relic Plugin</h2> The New Relic RPM service connects to your application via a plug-in that they provide, called the agent. Installing it is as simple as for any other plug-in: <pre> script/plugin install http://svn.newrelic.com/rpm/agent/newrelic_rpm </pre> There’s two modes in which RPM can run. In development mode, it runs on your local development server. In either mode, the plugin hooks into the Rails framework to automatically collect performance data. In Development mode, you examine the performance data RPM collects via views that it installs into your application. In this mode, RPM stores only the last 100 HTTP transactions, and it keeps all the analysis data in memory, so it doesn’t touch your database. In production mode, it runs on your production server and sends data to New Relic’s server once per minute. New Relic’s server then provides performance reports that you can view via the web, without impacting your production server. In the screencast, we explore the basic features of the New Relic system, and then dive into a specific performance optimization example. For details on New Relic’s system, check out their <a href="http://sites.google.com/a/newrelic.com/rpm-docs/">documentation</a>. We’ll start with development mode, and then take a look at a real application running in production mode. <h2>Exercise the Application</h2> With the plug-in installed in our sample application, we’ll start the server: <pre> script/server </pre> And then browse to <code>localhost:3000</code>. So far, this should look very familiar. Before we look at the analysis screens, we need some data to analyze. So let’s click around to several of the pages, and then log in and go to the admin interface. <h2>Viewing Performance Data in Developer Mode</h2> Now, to view the performance data, just browse to <code>localhost:3000/newrelic</code>. In doing so, you’re accessing a controller and views that the New Relic plug-in installed. The data is coming from the in-memory cache of information about the most recent 100 controller actions. The main RPM Developer Mode page lists the URLs for each of these actions, and the total time taken to complete each one. As you can see, nothing in this application takes very long, as you’d expect—there’s nothing here that’s doing very much work. Click on one of the URLs, and you’ll see a pie chart of where the time goes for that action. In the case of one of our simple CMS pages, about half the time is in the controller action, and you can see how the remaining half of the time is split among the rendering, database, and Active Record code. Below the pie chart, you’ll see the items listed out; click the question mark next to any of them to see a description and some tuning hints. Click the Details link to see a time-sequential breakdown of the components. The Duration column shows the total time spent in a particular method, including time spent in any other methods called by the selected method. The Exclusive column subtracts off time spent in framework code called in the method, such as ActiveRecord methods. For each line that involves a database query, you can click on the SQL link to see the query that was produced, as well as the stack trace that shows the related code. <h2>Running in Production Mode</h2> In production mode, the agent doesn’t accumulate data in memory; it sends it to New Relic’s servers. You then log in to the New Relic application, running on their servers, to view your data. New Relic has recently released a free version of its production mode service, called RPM Lite. It is limited to a couple of views, and to the last 30 minutes of data, but you can use it on any number of servers with any number of hosts (previously, the free plan was limited to a 90-day trial period and two mongrels). On a paid plan, which is priced starting at $40 per host for the Bronze subscription, you can view data from the last 7 days and you get many more views. The premium plans provide more ways to analyze the data. The Silver and Gold plans provide longer data retention of one and three months respectively and a bunch of advanced views and features, including transaction tracing, automatic incident detection, error tracking, day to day and week by week comparisons, and scalability analysis. To use RPM in production mode, you need to get a license key from New Relic, by signing up for either a paid or a free account. Your confirmation email will include a <code>newrelic.yml</code> file, which you should put in your config directory. There’s a variety of other settings you can tweak in that file. (You can also just copy your key from the New Relic site and paste it into the <code>newrelic.yml</code> file that the plugin installs.) With the agent and <code>newrelic.yml</code> file installed in your application, deploy it to the production server. That’s all it takes to begin collecting performance data. <h2>Viewing Production Mode Performance Data</h2> In production mode, there’s much more information available. Since it’s accumulated in New Relic’s database (currently at the rate of more than 10 billion reports a month for all customers combined), you can view data from a relatively long period of time. And the ways you can drill down into the data are much more flexible than in developer mode. Browse to <code>http://rpm.newrelic.com</code>, and log in with your (free or paid account) user name and password. In the screencast, we look at the live performance data for BuildingWebApps.com, and then show how we track down a missing database index, as well as the need for more caching, and then make changes to the code and verify that the performance did indeed improve. You’ll have to watch the screencast to see this information, since it is not publicly accessible. The home page lists each of the servers associated with the account. In our case, we have two production slices and one staging slice. You can define server clusters, so the production data all appears together. This “stoplight” view shows you at a glance if everything appears to be running ok. If all the lights are green, it’s good. Yellow means something is getting marginal, and red means trouble. You can define the thresholds for these color levels by clicking the “traffic light” icon next to the host or cluster name. You can click on any of the numbers next to the lights to go to a detail page. We’ll start by going to the overview page for the production server, by clicking on the Production label. From the strip at the top of any of these pages you can choose (from the three pop-up menus) which server or cluster you want to look at, which view you want, and what time period you want to examine. You can choose a period as short as 30 minutes, which lets you see a data point per minute, or extend a period as long as the last 7 days (one month for the Silver plan and three months in the Gold plan). These graphs show that our application’s CPU, memory, and database usage levels, as well as the response times, have been relatively stable, with modest fluctuations. On the Lite plan, there’s just one more view available. The Controllers view shows the overall throughput and response time graphs, plus a list of the slowest 20 controller actions during the time interval being shown. <h2>Advanced Views</h2> At the Bronze level (the least expensive paid version), you get a bunch of additional views: <ul> <li>Page Volume: Shows the number of views for the 15 most frequently viewed URLs during the selected interval</li> <li>Active Record: Shows the total time spent in the database for find, save, and destroy operations, the database response time, and actions that are the top 15 database consumers</li> <li>Index Hunter: Shows the slowest Active Record queries, which are good places to look for missing indexes</li> <li>Time Consumption: The top 15 consumers of time — a good way to find the places where optimization would have the biggest impact</li> <li>Cluster Breakout: How performance varies for each cluster and each Mongrel</li> <li>Compare with Yesterday: Are things better or worse than yesterday?</li> </ul> The Silver levels adds: <ul> <li>Transaction Tracer: Lists all HTTP transactions that took more than two seconds; click on one to drill down into the details</li> <li>Errors: All Rails errors, nicely aggregated; click one for details</li> <li>Incidents: List of metrics that were more than 50% worse than for the same time period one week ago</li> <li>Week-to-Week Comparison: Compare summary metrics against one week ago</li> </ul> And the Gold level adds: <ul> <li>Scalability Analysis: Plots performance vs. application load</li> <li>Response Time Report: A configurable report that shows the sources of response time degradation</li> </ul> <h2>Tracking Down a Performance Issue</h2> To see an example of a performance problem being tracked down and corrected, watch the screencast (right-click the button at the top left to save the video file to your computer). The tasks we demonstrate on the live BuildingWebApps.com code include: <ol> <li>Identifying when too many transactions are happening (lots of small calls to our DB adding up). We proceed to implement a caching solution to reduce the number of calls.</li> <li>Identifying longer duration queries that are sometimes caused by lack of proper indicies on relevant database tables. We apply an index and see our queries change from traversing through many DB rows of data to near constant time access.</li> </ol> We give a demonstration of the MySQL EXPLAIN command, and then show that the same information is available within NewRelic in both development and production modes.

23: Performance Analysis with the New Relic RPM

Directory

Click for all Categories