Production Deployments Don't Have to Be a High Wire Act
By Dalibor Siroky
It's 3 am in California, and you are still awake trying to sort through a release process that has been delayed by several hours. The deployment to the main application cluster took several extra hours due to an unanticipated problem with the servers, and now everyone is waiting on the lead database administrator to call into a conference bridge so you can all move on to Step 53.5b of the deployment. Your production deployments always seem to be problematic and this one might be the worst you've experienced yet.
Everyone's Tired: Bad Decisions Abound
By "everyone," I mean the 30 QA testers and 3 project managers in California, several developers in Sydney and India, and an operations team spread throughout the EU and the US. You can tell by some of the ambient noise on this conference call that people are starting to falter.
Your California team is up at 3am and exhausted to the point that they are making irrational decisions about risk and pushing changes without adequate testing. Some people on the bridge are yawning while others are just waking up. You work for an impressively global organization, but it's times like these that make you realize your release process needs to be refactored to allow for more sleep.
What Time Zone Is This Spreadsheet Using?
Your developers are attentive and well rested, but the deployment playbook Excel spreadsheet lists a sequence of events to be completed with times like 11 PM and 10 PM. "10 AM shift all traffic to backup database. 10:30 AM upgrade database. 11:00 AM deploy new code to application servers."
There's no mention of time zone in this spreadsheet which means that the Sydney team is constantly asking, "This release plan is in PST, right?" You run a global company, but no one bothered to add time zone to the release plan; unfortunately, this caused problems when the team in India decided that "10 AM" meant "10 AM Bangalore time." Twelve and a half hours before the release was even scheduled to start, your production database was taken offline and upgraded. You make a mental note to yourself - "specify time zones on all release spreadsheets."
"No one else can run Step 53.5b, really? Can someone call Jack?"
Six months ago you scheduled an emergency meeting to convey with a sense of urgency that every project must have a repeatable, automated release process. Most of the teams responded quickly: the application team perfected a series of scripts to automate deploys, but the DBAs are another story. Six months later you have a DBA on the phone informing you that no one other than the lead DBA - Jack - can run the necessary scripts to modify replication settings in production.
You make another note - "Tell DBA team manager to ensure that all production deployments have coverage." Tired and annoyed you tell the team, "Ok, can someone get Jack on the phone so we can move forward?" Minutes turn into hours. At 4AM your team reports that Jack isn't picking up his phone and someone remembers that he might just be camping in the middle of the Australian outback. There's no wireless signal within miles of Jack so your entire release now depends on seeing if the intern can figure out what steps Jack would have performed on your production database. Everyone is blocked.
Time to Rollback?
You are beginning to have that sinking feeling that you might have to tell everyone to execute the rollback plan. That's not something you want to have to start unless you have to, but it's 4 am and your deployment window is going to end. There's one problem: no one bothered to update the rollback plan so you are not even sure it's possible.
A rollback also requires the input of management...
Another Broken Production Deployment? How Surprising.
The VP called into the bridge to check on status and made a sarcastic comment about how surprised he was that we were having problems. He wasn't really. He's used to production deployments reflecting poorly on his leadership skills. In that sarcastic remark is a hidden message to you: fix this or I will find someone who can and I'm not waiting much longer.
This happens every time your team does a release, and you keep on having to explain it away to management as something that comes with the territory of running a "complex system." The only reliable constant in your organization is that your release process causes serious downtime, and after months of effort it still isn't getting better.
Your valid excuse consists of pointing to the tools used to manage the release process - a combination Excel Spreadsheets, wiki pages, and Remedy tickets. You tell your management that teams can't communicate with each other because there's no coordination between systems. One team might be focused on JIRA and Rally while the other might be focused on BMC Remedy. Instead of having one tool to hold teams accountable your team ends up starting 120 person email threads and you are sure several teams don't read your release announcements.
A Hundred Wikis with an Excel Spreadsheet Playing Catch-up
The problem isn't entirely tool-focused, but there's no reliable way for your teams to surface management issues and other problems in a transparent way. Your broken deployment process is a symptom of a larger issue with. You lack a consistent approach to deployments because your deployment process is spread across a hundred Wiki pages, fifty development teams, and a high-level Excel spreadsheet that is always playing catch-up.
It doesn't have to be like this. Your deployments don't have to be a high-wire act that always seems to end in one of your systems failing in production.
With Plutora's Deployment Manager you can keep track of your deployments alongside the releases and environments they are designed to orchestrate. You can identify the people responsible for each step and ensure that everyone understands exactly what schedule is happening when. You'll have a single source of truth for release status so that you don't have to spend 40 hours on the phone asking, "What step are we on now?"
If you are looking for release play books that can be measured and evolved over time it's time to start using Plutora.
The post Production Deployments Don’t have to be a High Wire Act appeared first on Plutora Inc.
@DevOpsSummit at Cloud Expo taking place June 6-8, 2017, at Javits Center, New York City, and is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
DevOps at Cloud Expo / @ThingsExpo 2017 New York
(June 6-8, 2017, Javits Center, Manhattan)
DevOps at Cloud Expo / @ThingsExpo 2017 Silicon Valley
(October 31 - November 2, 2017, Santa Clara Convention Center, CA)
Download Show Prospectus ▸ Here
The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
@DevOpsSummit will expand the DevOps community, enable a wide sharing of knowledge, and educate delegates and technology providers alike. Recent research has shown that DevOps dramatically reduces development time, the amount of enterprise IT professionals put out fires, and support time generally. Time spent on infrastructure development is significantly increased, and DevOps practitioners report more software releases and higher quality. Sponsors of @DevOpsSummit will benefit from unmatched branding, profile building and lead generation opportunities through:
For more information on sponsorship, exhibit, and keynote opportunities, contact Carmen Gonzalez by email at events (at) sys-con.com, or by phone 201 802-3021.
The World's Largest "Cloud Digital Transformation" Event
@CloudExpo / @ThingsExpo 2017 New York
(June 6-8, 2017, Javits Center, Manhattan)
@CloudExpo / @ThingsExpo 2017 Silicon Valley
(Oct. 31 - Nov. 2, 2017, Santa Clara Convention Center, CA)
Full Conference Registration Gold Pass and Exhibit Hall ▸ Here
Register For @CloudExpo ▸ Here via EventBrite
Register For @ThingsExpo ▸ Here via EventBrite
Register For @DevOpsSummit ▸ Here via EventBrite
Sponsors of Cloud Expo / @ThingsExpo will benefit from unmatched branding, profile building and lead generation opportunities through:
For more information on sponsorship, exhibit, and keynote opportunities, contact Carmen Gonzalez (@GonzalezCarmen) today by email at events (at) sys-con.com, or by phone 201 802-3021.
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Track 1. FinTech
Track 2. Enterprise Cloud | Digital Transformation
Track 3. DevOps, Containers & Microservices
Track 4. Big Data | Analytics
Track 5. Industrial IoT
Track 6. IoT Dev & Deploy | Mobility
Track 7. APIs | Cloud Security
Track 8. AI | ML | DL | Cognitive Computing
Delegates to Cloud Expo / @ThingsExpo will be able to attend 8 simultaneous, information-packed education tracks.
There are over 120 breakout sessions in all, with Keynotes, General Sessions, and Power Panels adding to three days of incredibly rich presentations and content.
Join Cloud Expo / @ThingsExpo conference chair Roger Strukhoff (@IoT2040), June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA for three days of intense Enterprise Cloud and 'Digital Transformation' discussion and focus, including Big Data's indispensable role in IoT, Smart Grids and (IIoT) Industrial Internet of Things, Wearables and Consumer IoT, as well as (new) Digital Transformation in Vertical Markets.
Financial Technology - or FinTech - Is Now Part of the @CloudExpo Program!
Accordingly, attendees at the upcoming 20th Cloud Expo / @ThingsExpo June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA will find fresh new content in a new track called FinTech, which will incorporate machine learning, artificial intelligence, deep learning, and blockchain into one track.
Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses.
FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.
More than US$20 billion in venture capital is being invested in FinTech this year. @CloudExpo is pleased to bring you the latest FinTech developments as an integral part of our program, starting at the 20th International Cloud Expo June 6-8, 2017 in New York City and October 31 - November 2, 2017 in Silicon Valley.
The upcoming 20th International @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA announces that its Call For Papers for speaking opportunities is open.
Submit your speaking proposal today! ▸ Here
Our Top 100 Sponsors and the Leading "Digital Transformation" Companies
(ISC)2, 24Notion (Bronze Sponsor), 910Telecom, Accelertite (Gold Sponsor), Addteq, Adobe (Bronze Sponsor), Aeroybyte, Alert Logic, Anexia, AppNeta, Avere Systems, BMC Software (Silver Sponsor), Bsquare Corporation (Silver Sponsor), BZ Media (Media Sponsor), Catchpoint Systems (Silver Sponsor), CDS Global Cloud, Cemware, Chetu Inc., China Unicom, Cloud Raxak, CloudBerry (Media Sponsor), Cloudbric, Coalfire Systems, CollabNet, Inc. (Silver Sponsor), Column Technologies, Commvault (Bronze Sponsor), Connect2.me, ContentMX (Bronze Sponsor), CrowdReviews (Media Sponsor) CyberTrend (Media Sponsor), DataCenterDynamics (Media Sponsor), Delaplex, DICE (Bronze Sponsor), EastBanc Technologies, eCube Systems, Embotics, Enzu Inc., Ericsson (Gold Sponsor), FalconStor, Formation Data Systems, Fusion, Hanu Software, HGST, Inc. (Bronze Sponsor), Hitrons Solutions, IBM BlueBox, IBM Bluemix, IBM Cloud (Platinum Sponsor), IBM Cloud Data Services/Cloudant (Platinum Sponsor), IBM DevOps (Platinum Sponsor), iDevices, Industrial Internet of Things Consortium (Association Sponsor), Impinger Technologies, Interface Masters, Intel (Keynote Sponsor), Interoute (Bronze Sponsor), IQP Corporation, Isomorphic Software, Japan IoT Consortium, Kintone Corporation (Bronze Sponsor), LeaseWeb USA, LinearHub, MangoApps, MathFreeOn, Men & Mice, MobiDev, New Relic, Inc. (Bronze Sponsor), New York Times, Niagara Networks, Numerex, NVIDIA Corporation (AI Session Sponsor), Object Management Group (Association Sponsor), On The Avenue Marketing, Oracle MySQL, Peak10, Inc., Penta Security, Plasma Corporation, Pulzze Systems, Pythian (Bronze Sponsor), Cosmos, RackN, ReadyTalk (Silver Sponsor), Roma Software, Roundee.io, Secure Channels Inc., SD Times (Media Sponsor), SoftLayer (Platinum Sponsor), SoftNet Solutions, Solinea Inc., SpeedyCloud, SSLGURU LLC, StarNet, Stratoscale, Streamliner, SuperAdmins, TechTarget (Media Sponsor), TelecomReseller (Media Sponsor), Tintri (Welcome Reception Sponsor), TMCnet (Media Sponsor), Transparent Cloud Computing Consortium, Veeam, Venafi, Violin Memory, VAI Software, Zerto
About SYS-CON Media & Events
SYS-CON Media (www.sys-con.com) has since 1994 been connecting technology companies and customers through a comprehensive content stream - featuring over forty focused subject areas, from Cloud Computing to Web Security - interwoven with market-leading full-scale conferences produced by SYS-CON Events. The company's internationally recognized brands include among others Cloud Expo® (@CloudExpo), Big Data Expo® (@BigDataExpo), DevOps Summit (@DevOpsSummit), @ThingsExpo® (@ThingsExpo), Containers Expo (@ContainersExpo) and Microservices Expo (@MicroservicesE).
Cloud Expo®, Big Data Expo® and @ThingsExpo® are registered trademarks of Cloud Expo, Inc., a SYS-CON Events company.