Code Freeze 2020 - Observabillity

Sorry for the loose nature of the notes rather than a good writeup, but I wanted to get things collated so I can work with them.  Good conference.  I ate at Al's for breakfast and Hong Kong Noodles for lunch.  And I realized I take the green line over to the U of MN for lunch anytime I'm downtown at work, which didn't occur to me these last six months.  If I had my bicycle, it's even a short ride that way across the campus bridge.  I need to get my urban on.  Not as much practical knowledge at this one (for me) as some of the past events, but that means I can focus on the few things I think have practical value rather than being all over the place.

Observability and the Glorious Future - Charity Majors (Honeycomb.io)

• O'Reilly Database Reliability Engineering (November 2017: http://shop.oreilly.com/product/0636920039761.do)
• How often do you deploy.  How long, how often do you fail, recovery time - the basics.
• Hires for communication skills (initial tech interview is to get them talking at the in person).  "Empowered to do their jobs". 
•  "How do I know if it breaks?" - all changes, all features
• "Serverless was a harbinger.  Deployless is coming."
• Developers (senior+) should amplify the hidden costs.
• Team happiness = customer happiness (Steve says this too)

Observability in Big Analytics - Bonnie Holub, Teradata

• Focus on the destination.
• VALUE
• Use what's out there (KDNuggets.com, Gartner)
• Predicts 2020:Analytics and BI Strategy - Gartner (behind a paywall, I've asked about a Gartner sub at VP before.  Can potentially get one via a third party, but I don't know if there's bias/editing: https://www.gartner.com/en/documents/3978987/predicts-2020-analytics-and-business-intelligence-strate)
• Ditto:  https://www.gartner.com/en/documents/3891788/15-insights-for-managing-data-science-teams
• Also likes:  https://www.mckinsey.com/business-functions/organization/our-insights/unlocking-success-in-digital-transformations
• Talked about Lighthouse Projects - this is what Healthy Habits Recommender was/is.
• There aren't just 5 elements of data science there is also:
•     Discover >> Access >> Prepare (wrangle, about where 85% of companies are and spend their time) >> Create models >> Socialize Results (dictionary as well) >> Deploy models >> monitor models >> analytics workflow
• (Look for her slides to - there is one for the last point that was particularly good)

50 Years of Observability - Mary Poppendieck

• What is the equivalent of metal fatigue in software?  Operator fatigue. >> e.g. what Steve pushes that a focus on PIs is important.
• Talked planes, bridges, three mile island
• She likes the Control series by Brian out on Youtube....they're deep: https://www.youtube.com/channel/UCq0imsn84ShAe9PBOFnoIrg
• Observable - all critical states known from system outputs
• Observable is at war with complexity.
• Controllable activator - sensor can get back to a set state in a set time.
• If it's not observable, can it be totally controlled? (no)
• Fault Tolerance: replication and isolation.
• Responsibility (and understanding the big picture) leads to desire for observability (and isolation/duplication). >> PLEX team at VP is a form of big picture.

What's Happening in Your Production Data and ML Systems  - Don Sawyer, PhData

• Most practical of the lectures.
• Focus on decoupled systems: Data warehouse, ML Models.
• Talked Provenance as both origin and change over time.
• Timestamp everything UTC (use Google Time API as an example to change it during compute).
• Focus on: audit trails, data quality, repeatability, added info (pipeline).
• Metadata payload.  PROCESS: id/version, start/end, transformations, inputs, configuraitons, DATA VERSIONS: traces of issues, data change history, defect data, LINEAGE: sources, frequencuu of read.
• Last point was a little messy (from me) but you want to trace right down to the node data touched in transit so you can hydrate anything from the last known good state.
• NOT ALL DATA RECORDS require granular povenance.  Can be expensive (so much data).  Use a flexible or generic schema.  Don't use S3 (slow).  Storage considerations.
• Storage: 1.) attach info to the record (can get big, note that Avro and Parquet are meant to do this), 2 send a separate event message - separate provenance API, 3.) only track some.  Note that for API approaches you may end up going down a rabbit hole of tracking the tracking api.
• Alternatives: Amundsen (Lyft), Marques (WeWork), DataBook (Uber), DataHub (LinkedIn)
• Look at Apache Nifi (there's a pluralsight class)

Evolving Chaos Engineering - Casey Rosenthal, Verica

• Ships, shoes, fruit (apricots), helium mining.  He's a very funny guy.
• LOOK FOR  A VIDEO to watch with the team:  https://www.youtube.com/watch?v=JfT9UxcEcOE
• Principlesofchaos.org
• Reversibility: blue/green, feature flags, ci/cd, agile to waterfall.
• Moved responsibility away from the people who do the work (hierarchy)
• Myths:
• 1. remove the people causing the accidents.
• 2. document best practices and use runbooks.  (most interesting problems are unique)
• 3. defend against prior root causes, aka defense in depth.  Root cause analysis: "at best, you are wasting your time."  Was our sponsor audience issue an example?  The answer was in part to restrict audience size.  But the dig highlighted system no longer supports system-wide features after growth, high processing cost of feature, inability to test with all users, etc.
• 4. enforce procedures
• 5. avoid risk
• 6. simplify
• 7. add redundancy
• Do NOT eliminate complexity.  Navigate it.  CI, CD, CV - continuous verification (here's a link to a CV article:  https://thenewstack.io/continuous-verification-the-missing-link-to-fully-automate-your-pipeline/).  That's New Relic for us.
• Has two books: Chaos Engineering and Learning Chaos Engineering.  First book comes out June 2020.

Catch Up - Part IV - Code Freeze 2015

Code Freeze had a security focus this year.  And although much of it had to do with things, the lessons were still good.  I know I've been talking about Stephen Checkoway's presentation on how to hack a car to my team for months.  He and Bruce Shneier both had great presentations.  The break outs about bitcoin and secure email were a little less inspiring, but gave me time to go have lunch with Ming at the local Asian restaurant.  Checkoway's takeaway is that cars were built with the expectation that as long as you were within the firewall/existing system you were safe.  That led to buses being connected to the extent that if you could hack the radio or force a buffer overflow via a CD or get a car outside the 4G network so that it had to communicate via traditional phone, you could hack everything right up to the point of driving it remotely (horn, shut it down, listen if there's a phone, unlock the doors, start it, etc).  It's a good lesson if you trust your database to serve up clean data all the time just because it's your database.



The panel at the end was good and elucidated on some of what Shneier was discussing earlier in the conference.  Shneier doesn’t like the worst case approach in security.  There’s too much of that in his opinion (operating in the moment), although he recognizes it’s somewhat at odds with his message around considering your adversary in advance, and it leads to “muddling through.”  That’s an old rules approach and “the old rules don’t apply.”  Checkaway agrees and sees it as positive that there’s been a shift from something might happen, to “something will happen and we have to deal with it.”

Bruce Shneier noted that getting companies to share their security data is valuable, but not easy (companies worry about their reputation).  Sharing virus information and other information helps many other industries and initiatives.  I include a few of my notes below.  Mostly for me:

• Systems that support people instead of replacing people involve reduced switching costs (example, the military).
• Speed is the advantage in an iterative loop of the hacker versus the hacked.
• Observe, orient, decide, act (OODA)
• Detection = IT.  Response = People.
• So improve them, training them, use the OODA loop, consider the switching cost.

1. We are losing control of our IT infrastructure.”  This means we’re also losing visibility.  We are also using devices we have less visibility/control over.
2. Attacks are more sophisticated in their skill and focus.  There is relative security and absolute security.  “A sufficiently motivated attack will get in.  Full stop.”  The attacker has the advantage.
3. There is an increase in government involvement in cyberspace, pro and con.  Some countries are selling cyberweapons (as arms dealers; turnkey solutions).

• The economy affects security:
•  Switching costs (WP vs. Word)
•  Managerial costs
•  Fixed costs (stamping out more copies)
•  Lemons Market (cheap and easy wins and that’s not always good)
•  Risk seeking when it comes to losses, risk adverse when it comes to gains.
• “People will drive to work and fear terrorism.  What are you thinking?”
•  Security is hard to sell.  Management is willing to “take the chance.”  Think burglar alarms  Car alarms
•  Look up Asian Disease Experiment on Wikipedia: http://en.wikipedia.org/wiki/Framing_effect_%28psychology%29
•  Look up his Crypto-gram newsletter.
• There’s an MSST degree in security at the U of MN.

Code Freeze 2012 Continuous Delivery - II

This summary is not available. Please click here to view the post.

Code Freeze 2012 Continuous Delivery - I

I went to Code Freeze 2012 at the U of MN this week.  I've been going for years and although some years are better than others, I generally walk away with something worthwhile.  This year's topic was "Continuous Delivery", a subject I'm very interested in as, before I was moved off the desktop products I was managing, I was trying to get them up to speed with Continuous Build and Continuous Delivery.  There were almost 13 separate products without unit test that were deployed to the desktop: e.g. 64 bit windows, 32 bit windows, Windows 7, Windows XP, Windows Vista, Office 32 bit, Office 64 bit, Office 2007, Office 2010, and a variety of other configurations.  One of the more amusing days was when a customer called to complain that our product didn't work quite right under Parallels.  As if we'd tested that configuration given we weren't testing anything other than manually over a two week period.  Embarrassing.

The conference was better than usual, with presentations by Jez Humble of Thoughtworks, Michael Nygard who wrote Release It!, Jenna Pederson (can I say cute local developer and not get in trouble? She's more than cute...her brain is cute, as proven by her blog and, I could add, Jez Humble was good looking as well, and had a better accent...perhaps that protects me from developer sexism), David Hussman (he of the many "dudes"), and John Penix of Google.  It made me appreciate that no system was so legacy-ridden or large that it couldn't be refactored.

Jez Humble (not a consistent blogger) of Thoughtworks was not only informative, but incredibly funny.  He's a walking series of sound bites on legs.  One of my favorites was about Agile, "Everyone's taking orders standing up instead of sitting down."  Or, "We shouldn't have to come in and sit in a data center for the weekend.  Life shouldn't be like that."  Word.  And, after talking about a dev ops moment in which dev ops said "no thank you", he clarified, "They were operations people...they were not that polite."

Jez talked about lean software development, quoting Eric Ries, a proponent of a Minimum Viable Product.  For those of you in agile, this is the concept that you should always have working software.  And just enough.  I'm a huge proponent of this.  It's perhaps the most important aspect of agile to me.  If you have a project, then you should have short cycles where you provide something that works.  Might not be perfect.  Might not be what you want in the long run.  But it's a deliverable that if you were to put it in front of customers, it might make money.  It gives you a plateau where you can decide whether to continue at all, take a break, fire your developers and hire new ones...some location where you can make a decision and still have something in hand that is concrete.

He talked about flickr.com, where they do 10 releases a day.  When Yahoo picked them up, they tried to change them, but were changed instead.  As is appropriate.

He talked about Poppendieck's Implementing Lean Software Development and the idea that you can measure your product by the amount of time it takes to change one (1) line of code (pg. 59).  Based on the products I managed, 24-48 hours minimum, even if all the Directors and VPs are available.  That's too long. The line should change in 1-2 hours, and the build/test/deploy/test should take another 2 hours.  Max.

And Jez talked about how continuous delivery is composed of three major parts:

• Automation
• Patterns and Practices (the subject of his own book about continuous delivery)
• Collaboration

Sounds about right to me.  If you don't include the perspiration.

Real World Agile User Experience Design - Jeff Patton (Part I)

I listened to Jeff Patton at Code Freeze 2010. He comes more from the user experience side than (Alan) Cooper, and he presented on the integration of design into day-to-day work. His core idea is that Agile, in eliminating the old BDUF (Big Design Up Front - I like BUFD, it sounds more like "buffed", implying some sort of dim, Jersey Shore, over-muscled, guy, which I can correlate to the slowness and excess of process often associated with Waterfall. By the way, you can get your own Jersey Shore nickname here. Mine is "Tan Jovi") process, has fundamentally altered how design is accomplished. Newer User Experience and Interaction experts are increasingly relying on lighter, more collaborative, practices that make design part of a holistic project approach.

Patton says that this history can be broken down into four parts. 1.) How User Experience (UX) was. 2.) How Agile wrecked everything. 3.) How User Experience design is in an Agile context. And 4.) What's next.

How User Experience/Interaction Design was...

Design followed an isolated phase model. That is, design was a separate precursor to other work, done by a designer or Business Analyst or business representative up front. It was characterized by fewer people, less collaboration, and more specialization. The emphasis was on information gathering, documentation, completeness, defensibility, and culpability. We all know the model or some variation on the model. Business needs >> user needs >> model the users >> high level design >> detail design >> specification >> develop software >> test (test, test, test...test, test, test) >> release >> maintenance.

The isolation of phases was obvious in the language gap (or syntactical overlap - choose your poison) between design and development and, later, between legacy design words and Agile vocabulary. Consider the difference between old school design and Agile when it comes to phrases and words like design, iteration (Agile builds it, Design sketches it), customer (Agile focuses on user story providers, Design focuses on the person who buys the product), user story (Agile focuses on what someone wants built, and Design focuses on a profile of a user), test-driven, and a "small bit of software" (Agile means "built in a few days". Design/UX means "something that seems small as a piece of user functionality. There's a enormous gap when you give that some study). The result when Agile, with its holistic, iterative, design/development/testing entered the scene, was to provoke in UX designers what Craig Villamor referred to as the five phases: denial, anger, bargaining, depression, acceptance.

Snowman Model

At Code Freeze, Jeff Patton referred to the "Snowman Model of Iterations" where the daily scrum feeds into larger iterations (Powerpoint of the snowman here). He scribbled a picture much like the one below. There was quite a bit of focus at Code Freeze on being able to draw some of the ideas that are involved in Agile or your design, so that they make a lasting impression on the many players in the project, particularly the product owners. This isn't just an Agile skill. Being able to create a lasting image in the mind of your audience is one of the best ways to foster change. If you're in my group at work, you may have seen "The Heart of Change" (John P. Kotter) on a manager's bookshelf, or you saw the video at the organizational realignment meetings where managers sported mullets and talked about how the web was just going to be a fad and CDs were the future. Kotter's book recommends making that visual leap that makes people feel rather than see, with images akin to Patton's snowman. If Agile is about harnessing change, then it's also about creating that indelible image for your audience that communicates that vision and creates a repository of shared imagery.

What do Agile Managers Look Like?

Alan Cooper was asked at Code Freeze 2010 "What do Agile Managers Look Like?" His response was, a manager can't manage craftspeople. Instead, they have to facilitate. A good Agile manager will [and I'd posit any manager should]:

1. Define and reiterate the overarcing goal (focus on the quality line, not the deadline),
2. Promote standards,
3. Focus on what to avoid and potential obstacles,
4. Help everyone on the project focus,
5. Own the problems and eliminate the roadblocks.

The factory has moved to the brain, and managers who continue to follow an outdated factory floor management paradigm are going to have an issue with that change. Don't think you can convince those managers. Just do the right thing and drag them along so they have to confront reality.

Code Freeze - The Collective Groan

One of my favorite moments at Code Freeze 2010 was when a presenter noted that everyone pictured on a particular slide worked six different projects. At that pronouncement, a general groan arose from the audience.

Where I work, I currently keep a load of around half a dozen projects, although if you slice and dice them, or consider my space to overlap with that of the other manager who does the same work, the portfolio is considerably larger. Fortunately, I have smart leads, and they really handle the bulk of the lifting, allowing me to focus on wider issues. But to stress how many projects you can expect working in my area, one of the question sets on my lead hiring interview form is "Have you worked in a matrixed environment? On how many projects simultaneously? How do you handle the context switching and constant flow of information? Where do you put your focus when there are competing priorities?" I've dinged more than one interviewee for never having worked on multiple, large projects simultaneously. Despite the groan from the Code Freeze audience, the interview process turns up a significant number of developers who are allowed to maintain their focus.

In my last role, I think I maxed out at thirteen simultaneous projects, not including my generalized day-to-day role. One project, overseas, consumed about 50% of that time, and the others cycled up and down over days and weeks from zero to twenty hours. But all were partially active, somewhat different, and not predictably cyclical. It wasn't abnormal to have twenty hours of concrete work flow in over the course of a day. The goal was then to push it down as fast as possible because you couldn't be sure the downswing was coming immediately. Additionally, when you were on call, you might be fielding questions, requests, functionality assessments, architecture explanations (and changes), training, coding (yes, coding - I still touch code, even now), and more, for upwards of twenty simultaneous teams/projects (we called them "partners"). Handling half a dozen projects in my current role has been somewhat relaxing by comparison.

But I should append that it was my old manager who once took a look at my forty hour per week schedule and dumped an extra twenty hour per week partner on me, overloading me. She was the first person who ever concretely pointed out to me that I work more effectively when I've got a little too much to do. That sort of load compels me to find ways to streamline until I'm back down to forty, and the processes and code I put in place generally benefit everyone. It was a good call on her part, and I've always felt it's evidence she was really paying attention to me in her managerial role.

I guess I'm trying to say I wasn't one of the attendees who groaned - which is a bit suspicious, because as a developer I seem to thrive in a matrixed environment (I'll post a bit about big company hiring premiums later) - but knowing that each employee is different, I can empathize with those that did, particularly as agile recommends avoiding context switching, and find some humor in the general despair that it's often expected of those who thrive in a more focused environment.

Code Freeze 2010

Remember, if you see xb in the label, it may be cross blogged, and it's definitely about Agile or some aspect of development. Feel free to avoid it, although you will be missing out on a little deeper dig into some of what I do day to day.

Last Thursday, the day of the icy streets here in the Twin Cities, I attended Code Freeze 2010 at the University of Minnesota. In the past, topics have included Maximizing Developer Value, Innovation, and Global Systems/Global Development. This year's theme was Redesigning Agility.

After hearing Ming wasn't going and learning from him which developer was (apparently there was more than one from our department who went, but that information didn't make it down to everyone who was participating), I contacted Jonathan, determined he lived in Eagan, and arranged to be the designated driver. Considering it took 35 minutes to travel 35E from Lone Oak to the Minnesota River (a few miles) on the way to the conference, and an hour and forty-five minutes to get back to Eagan from the U of MN Alumni building, it was a good choice to carpool as it gave me someone to talk to for most of three hours of iced interstate driving. Maybe Jonathan disagrees. Maybe we didn't talk. Maybe he just got talked at. I seem to remember monopolizing the conversation.

The Symposium was great and I'll post some follow up on the presentations by Alan Cooper, Jeff Patton, David Hussman, and Tim Andersen. I appreciated that as a conference focusing on Agile, Interaction, and Design, and where the intersection of those activities is headed, the presenters felt there was space to disagree. Too often when I've gone to an Agile event, it feels as if all the presenters are lock step in their approach. That seems to be acknowledged by others as well, as there's plenty to read about whether Agile is a methodology or a belief system. Code Freeze had none of that feel.

A few non-Agile, conference-related items reminded me I live in Minnesota:

• When I had to go back to the parking ramp for my blackberry, I had windburn on my face before I'd even made it off the Alumni Building sidewalk.
• The steeply slanted windows on the Alumni Building pick up the snow in waves that flow down the windows in a cascading series of white akin to what you see when you shake an etch-a-sketch. It looks like you would expect the dune-building process at White Sands to look if you could accelerate time.
• So many people are bundled up that you can see bits of down floating on the air current inside, dancing in the sunlight. Minnesotans molt in the winter, just like geese. It's probably one of the reasons I sneeze so much, and my eyes water, at random times (that aren't so random if you listen for the blower).
• Despite those arctic-related anecdotes, there's still a constant stream of bicyclists passing by the window. Some without a hat or a jacket.

Code Freeze 2009

I just noticed this in my drafts. I started writing up some details around Code Freeze, and then just left it sitting there. For some reason, I thought I had posted it. I'm only posting it now so it gets indexed and I don't forget the books/links I was interseted in from the conference...

Last week, on a day so cold we stayed in the sun while walking to lunch to avoid sticking to the sidewalk, a number of us from work attended the University of Minnesota Code Freeze 2009. In the past there have been a few coworkers there, but this year there was a complete contingent, including two coworkers who were coordinating the conference, folks from my department, folks from Findlaw, alumni of Findlaw, a folk from TTA, and others from the content area.

I've been to two other Code Freeze events in the past and the scope of the topic is generally pretty broad "Innovation" in 2008 (with a lot of Agile talk), "Global Systems, Global Development" in 2007 (with a lot of offshoring talk), and "Maximizing Developer Value" this year (a mix of how not to get interrupted, agile, estimating, testing and the odd other topic).

Neal Ford spoke on "On the Lam From the Furniture Police". I thought he was by far the most interesting speaker and his presentation was enjoyable with a good sense of humor and specifics about how to avoid "context-switching" as a developer, that problem we've all had that we just get going on code when someone drops by to talk about something completely unrelated. It's not just a developer problem. It happens to everyone. It even happens to managers, as I can attest in my recent role trying to coordinate two distinct workstreams that kept stepping on each other. When I worked with our large, priorprietary database, I worked with a dozen partners at once and seldom had issues with flipping from project to project (to project to project) even within the same few minutes. However, when I was doing the role of a data analyst and my standard role earlier in that job, when I had much less work overall, switching between the two roles often threw me for ten or fifteen minutes each time I moved from one to the other. It's interesting that context switching can be context specific.

Ford also talked about the value in engaging both sides of your brain, the logical and the holistic (intuitive, call it what you will - the side that says "aha" about problems). He then related this back to context switching by pointing out that it took a lot of work to get your right side engaged while coding, and context switching was death to your aha moments.

So what does Mr. Ford offer as ways to reduce context switching and engage right brain thinking? Toys - provide some little toys and brain teasers so that the left side of the brain is engaged in a repetative activity allowing the right side to override the noise. Get rid of "pin prick" distractions. Turn off your notifications. Use a tool like Tweak UI to turn off balloon tips. Employ a screen dimmer like Jedi Concentrate that ensures the window you're working with garners your attention. Use tools to track relationships like an external brain (he called it an exo-cortex): a notebook, Personal Brain (perhaps useful for writing a novel as well), Larry's Any Text File Plugin (in conjunction with Google Search), and using Rooted Views.

Use automation and continous integration to cut down on unnecessary distractions from your own code. Use virtual desktops to mimic how you would lay out papers on a larger desk. Have walking meetings. Exercise. Take a nap at 3:30 each day. Don't work in a cube. Avoid sitting where the noise is heterogenous instead of homogenous. I see this last one in effect at work almost every day. We have developers who work on a particular type of project and they seem very capable of flipping between those projects and avoiding the context switching associated with many projects, although we have initiatives to try and ensure that any one person is on a minimal number of projects for that reason. However, in the same area are developers who are on completely different types of work, whose day-to-day worries are more along the troubleshooting line, tackling a problem of the day, or a tricky hardware issue. When they meet to discuss the issue of the day, it can suddenly engulf everyone nearby, even developers from other areas who can't avoid the heterogenous noise.



Books mentioned:
Andy Hunt: Refactoring Your Wetware
Pragmatic Thinking and Learning
Peopleware: Productive Projects and Teams
Raston - Human Interface (locus of attention)
Flow: The Psychology of Optimal Experience
The Productive Programmer

Nate Schutta - Hacking Your Brain for Fun and Profit
Nate was the second presenter, and the only other one I want to talk about. Susan Standiford, Andy Miller, Tomo Lennox and Luke Francl were interesting, but they weren't on topic with anything that resonnated with me. Nate continued Neal's presentation in some respects, focusing on what there was in the brain of a developer that lent itself toward productivity. He focused on some of the physiological effects of not sleeping enough, not getting enough exercise, email apnea, and continuous partial attention. He offered up solutions such as no-meeting Fridays, f-off flags (in use at TR in a few places), and a zero inbox. All great ideas. While talking about the importance of sleep, he referenced these amusing videos of Peter Tripp staying awake for 8 days.

Part I - http://www.youtube.com/watch?v=mXrANL9aqz8
Part II - http://www.youtube.com/watch?v=2R8jNJFFzS0&feature=related

Books mentioned:
Brain Rules
A Whole New Mind
Mind Hacks
Your Brain: The Missing Manual
Lifehacker (GTD)
Getting Things Done
Pragmatic Thinking and Learning
43 Folders (web site)