Robin Lee Powell

robinleepowell@gmail.com • (650) 296-8094
https://www.linkedin.com/in/robinleepowell

Linux Nerd, Finder Of New Talent

Professional Summary

I have been a sysadmin / DevOps / SRE professionally for over 20 years. For about 5 years I was also a manager. Off and on, in particular in the last 3 years or so (as of July 2023), I have also been a programmer / software engineer.

My summary of what I'm looking for in my next role is at a post I made on LinkedIn.

Particular things I'd like you to know about:

• As a manager at Apple, I was able to give about 6 people their first crack at a serious tech job, and most of them were hugely successful.
• I'm really good at building, automating, streamlining and debugging complex connections between disparate systems. A pretty good example is integration test frameworks, which tend to be at the center of many systems (virtualization, code build and deployment, the test automation system itself, etc).
• I really genuinely enjoy teaching, training, and levelling others up in anything technical.
• I am really good at debugging most things, but systems interaction problems in particular.
• I have a bizarre fondness for solving problems with the provisioning and early boot aspects of managing Linux servers, and have built systems to image/kickstart Linux hosts on an unreasonably large number of occassions.

Professional Experience

• After I moved out of a management position, I have been on a few different teams. My current team (July 2023) is responsible for low-level network security in our infrastructure.
• Despite the title, I usually end up doing systems integration / sysadmin / SRE work, because it's what I like and it's what I'm good at and I have yet to be on a dev team that *didn't* want "somebody" to write a better test framework of some kind.
• In my current role I've been learning a great deal about the deep internals of the Linux network stack, since my team is responsible for software that directly control individual packet flows to make sure they meet security policies.
• I am frequently one of the primary points of contact for internal expert user support. I work hard to establish trust with our customers, and to make sure they feel heard and taken care of.
• I got our network security software, which is very integrated into the basic functionality of the system, working on RHEL9, in two compute environments.
• I entirely rewrote our AWS-based integration test framework for my previous team to make it configurable to match arbitrary production environments.

• I was the manager of a tools team that provides infrastructure to other internal teams.
• I grew the team from 1 subordinate to a peak of 8, all while maintaining the team's quality of execution and strong focus on best practices.
• I did my own hiring and arranged my own interviewing. I am very proud to say that I gave several people their first chance at a serious tech job, and most of them were very successful. I'm also proud that my team was one of the more diverse in our part of the company at the time.
• My team's focus was on systems automation and configuration management; our primary internal product was Puppet backed by a home-grown CMDB, but we also provided DHCP, DNS, Kickstart, and other such "lights on" services for a very large pool of bare-metal systems in many datacenters worldwide
• My team was seen as the go-to team for internal discussions of SRE best practices, configuration management, and automation, a reputation we worked hard to build and of which we were justly proud
• The majority of my time was spent on communication between teams; there are something like 40 distinct teams and sub-teams that used my team's services, and one of the services we provided was L3 support for our tools, so the communication process was very nearly a full time job by itself. In this aspect of my role, I helped to grow my team's reputation for helpfulness and competence by leaps and bounds.
• A significant chunk of my time in this role was spent on recruiting and mentoring, where I specialized in finding high-potential junior candidates and rapidly bringing them up to our world-class standards. As of early 2021, 3 members of my team have been in tech roles for less than 2 years, and yet people are still coming to them for advice on an almost daily basis.
• As a people manager, I followed servant leader and leader-leader models: my focus was on the well-being of my team, both now and in the future, and their ability to learn and grow in their roles, rather than on short-term goals like simple retention. My feeling is that if people remember their time with me and our company fondly, then even if they move on to something else, they will provide a means to help me fill any positions that might need filling in the future.

• Hired as Senior Systems Administrator when the company was about 5 people; made Director Of Operations due to outstanding performance
• Hired additional talent to round out the operations team
• Entirely responsible for our ~40TiB of user data and ~500 AWS cloud servers (across 3 continents)
• Rebuilt our background task system to save ~$12k/month
• Transitioned the entire company's systems from leased servers to AWS, resulting in huge increases in user responsiveness (example: 5x improvement in background task startup time)
• Worked with the CTO to cut AWS costs by more than 30%
• Managed all systems-related vendor relationships; built strong relationships leading to significant discounts
• Effectively automated away the need for ~3 full-time system administrators, at vast savings
• Produced extensive documentation for internal use
• Trained all operations personnel, as well as provided extensive training for other departments on our internal tools
• Built disaster recovery plan from the ground up
• Worked extensively on security compliance (FISMA in particular)
• Built extensive web test automation in Selenium

• Was key in moving hundreds of servers from private to managed hosting, an extensive 8 month project
• Over several years, replaced a ~7 person team managing hundreds of servers with only myself and extensive automation, at vast savings
• Significantly improved backup behaviour and significantly reduced backup costs at the same time
• Managed vendor relations, particularly with our backup provider, through significant technical issues

• Personally responsible for all of the machines that ran www.furl.net (now defunct), which consisted of ~50 machines of ~6 functional types
• Part of a team managing hundreds of web search engine machines
• Single-handedly turned Furl from something that kept oncall up for hours 2 or 3 times a week to something that ran smoothly. This included extensive MySQL server optimization, query optimization, capacity planning, server buildouts, and various kinds of scripting, including using cfengine.
• Developed a system to take a machine from having no OS to being fully functional members of their cluster
• Performed near-zero-downtime software deployments
• On-call/pager duties; problem analysis and remediation

Volunteer Experience

I was the Treasurer/Secretary for the LLG for many years. I was responsible for the quarterly book-keeping and general financial management, as well as communication with other organizations.

I continue to provide both the server and network connection for the lojban.org infrastructure, at my own expense. Also, I support a wide complex of tools for a variety of projects.

Education

Bachelor Of Mathematics, University Of Waterloo, Canada