John A. Carter

DevOps Engineer / Cloud Architect
Boston, Massachusetts johncarter@warlordofmars.net (617) 334-5110 https://github.com/warlordofmars

An experienced and accomplished DevOps engineer and architect, with a background in both software development and operations, that has demonstrated a strong passion for automation and innovation of all facets related to the software development life-cycle and all underlying infrastructure required. Looking for a position that would leverage principles from software development, traditional systems administration, and other principles under the DevOps umbrella, in order to contribute to, and improve, an overall customer value chain of an organization that shares the same passion.

Experience

Principal DevOps Engineer

Cogito May 2015 - Aug 2018 Boston, MA http://cogitocorp.com

Served as technical team lead and cloud architect, developing a wide range of end-to-end automation and infrastructure solutions in an agile development environment whose focus was to build and maintain multiple siloed single tenant SaaS application environments (both Cloud and On-Prem), each consisting of a collection of microservices, storage, and databases, the purpose of which was to integrate with a corporate VoIP telephony system, and provide real-time analysis, feedback, and coaching for live phone calls as they occur.
  • Implemented standardized development environments using custom built images with Packer (Vagrant VirtualBox + AMI)
  • Developed masterless Puppet patterns to allow for management of all Cloud and On-Prem instances in a standardized manner, where all configuration for the instances are retrieved from encrypted values in AWS Systems Manager Parameter Store.
  • Developed standardized monitoring and metrics collection patterns on Cloud and On-Prem instances using Icinga, Diamond, Graphite, and Puppet
  • Defined standardized set of Continuous Integration patterns in use for all software projects, implemented via Gradle and Jenkins
  • Built full Continuous Delivery pipeline to test all services together, and promote into individual single-tenant environments (Cloud or On-Prem). Deploying infrastructure and application code simultaneously.
  • Developed Infrastructure as Code standardized patterns for all services using CloudFormation, making heavy use of Nested Stacks, allowing for the creation of a number of "sub-modules" that could easily be put together to build application components. Leveraging Lambda-backed CloudFormation Custom Resources for anything not already natively supported by CloudFormation.
  • Developed a number of custom Gradle plugins for standardized build, test, and deploy across all projects and services
  • Migrated legacy manually managed on-prem Jenkins instance to new fully automated Jenkins environment in the Cloud, with all jobs programmatically created via the job-dsl plugin.
  • Developed a number of internal ChatOps tools, allowing team to manage environments and load tests from within Slack
  • Designed and implemented a Multiple AWS Account architecture to isolate both production environments from lower environments as well as isolation between distinct business units of the organization.
  • Mentored other engineers in the ways of DevOps
  • Led migration effort from legacy On-Prem Pilot application deployments to full Cloud only deployments
  • Managed effort to scale up the application from single-instance deployment to enterprise-grade deployment of hundreds of instances in auto scaling groups behind ALBs.
  • Developed reusable patterns for deploying Asterisk PBX for local development and integration testing
  • Created repeatable patterns for creating AWS VPN Gateway connection between a customer On-Prem environment, and the customer's siloed Cloud environment
  • Using Packer and Puppet, created automation for building new custom AMIs for each service. Software releases consisted of updating AMI in Auto Scaling Group.
  • Implemented CloudWatch Logs, CloudWatch Alarms, and CloudWatch Metrics across all instances, Cloud and On-Prem, for unified metrics, monitoring, and logging across all environments
  • Built a proof-of-concept to migrate all of our services from Auto Scaling Groups to container-based deployments. Utilizing Gradle, created automation to deploy full stack local development environment using Docker compose, as well as deploying the full stack to ECS.
  • Participated in employee-managed supply of small-batch homebrewed Nitro Cold Brew coffee. Including brewing the coffee, cleaning equipment, and maintaining the nitrogen supply lines and taps.
  • Developed a custom serverless service to manage all new tenant creation, as well as subsequent updates to tenant environment. Utilizing AWS Lambda and AWS Parameter Store, clients of this service include a Python CLI, a Slack integration, as well as from calls within a Jenkins Pipeline.
  • Architected all solutions to conform with compliance requirements enforced by PCI and HITRUST. Helped facilitate organization certification for PCI and HITRUST.
  • Led effort to design and build repeatable deployments of highly customized AWS Aurora RDS MySQL and PostgreSQL clusters through CloudFormation, RDS Parameter Groups, and AWS Lambda backed custom CloudFormation resources
  • Architected and implemented a serverless yum package management repo system using AWS S3 and AWS Lambda capable of consuming upstream repository packages as well as hosting internal packages. Pipeline to promote packages through dev, test, stage, prod versions of the repos as they've passed acceptance.
  • Built a re-usable deployment of a full ELK stack that could be deploy along with a single-tenant environment
  • Implemented an automated means of updating internal Route53 zone records for custom hostnames on EC2 instances using AWS Lambda and CloudWatch Events
  • Evangelized and spoke out throughout the organization for adherence to the AWS Well Architected Framework
  • Instituted the practice of "DevOps Office Hours", allowing the entire organization to come together in a cross-functional capacity to solve DevOps problems and get DevOps advice
  • Participated in the migration of build platforms from Ant to Gradle for several Java services
  • Designed and implemented patterns using AWS Systems Manager SSM Agent to automate tasks on both Cloud and On-Prem instances, standardizing all maintenance, management, and troubleshooting tasks between the two.
  • Built fully automated and repeatable deployment of an internal Artifactory server in EC2 for hosting all artifacts produced by internal build system
  • Facilitated VoIP telephony integration with multiple enterprise customers, each with unique architecture and integration points, often requiring packet capture and network analysis, occasionally requiring custom network design.
  • Implemented standardized SSO across all internal applications, AWS, and SSH logins with FoxPass and Okta
  • Devised and implemented a resource tagging strategy, applied automatically, for all Cloud and On-Prem resources, allowing easy look up and identification of resources coming from multiple single-tenant customer environments.
  • Architected and implemented a unique proxy solution allowing On-Prem resources in completely isolated networks be able to access not only other application resources in the cloud but also API access to services on the public internet.
  • Created all required IAM roles and profiles via reproducible templates, with least-privilege rules applied across the board, allowing for customer data isolation in a shared AWS account
  • Built a fully automated Zookeeper deployment which seeded all relevant information required from encrypted values AWS Systems Manager Parameter Store, allowing services to bootstrap themselves in a newly provisioned environment.
  • Implemented multi-level network protection through AWS Security Groups, as well as iptables rules on all instances, only permitting explicitly what was required for application to function, all applied in a fully automated manner.
  • Built a serverless backup scheduling service using AWS Lambda, CloudWatch Events, and AWS Resource Tags, that created snapshot backups of all EBS volumes, rotating out old snapshots based of of defined retention rules.
  • Implemented two-factor authentication on all SSH access using Duo.

Principal Systems Administrator

Hachette Book Group Dec 2010 - May 2015 Boston, MA http://www.hachettebookgroup.com

Led team of systems engineers providing infrastructure and application support for one of the "Big Five" US book publishers.
  • Architected and implemented a complete private cloud infrastructure solution utilizing HP, NetApp, and VMware
  • Migrated all servers from legacy fiber SAN to NetApp NAS, configuring several filers from scratch
  • Migrated all legacy applications and physical servers to private cloud infrastructure
  • Migrated several legacy applications and databases from AIX to RHEL
  • Implemented Samba winbind on all Linux hosts for Active Directory login
  • Optimized VMware, NetApp, Cisco Network for NFS traffic
  • Migrated public DNS services to MySQL-backed PowerDNS
  • Implemented standardized Wiki solution, with standardized documentation templates
  • Provided lead infrastructure support in effort involving transitioning off of mainframe-based application to more modern application framework hosted in private cloud infrastructure
  • Developed enterprise-level job scheduler responsible running key application essential to the business
  • Developed infrastructure dashboards for executives providing real time stats and uptime
  • Developed automated workflow for release of new builds to dev/test/stage/prod
  • Implemented configuration management throughout the environment using Puppet, Linux + Windows, ensuring standardized environments between dev/test/stage/prod
  • Implemented a custom Puppet ENC
  • Developed a custom application that served as global repository of aggregated information about all infrastructure elements (servers, virtual machines, storage, databases, networks,
  • etc.), dynamically populated by what is actually in the environment
  • Implemented Kanban style workflow which included specific checkpoints for elaboration and documentation
  • Developed repeatable pattern for automation of all internally developed applications
  • Highly optimized NetApp filers for optimal performance of high-load Oracle databases
  • Provided monitoring of all aspects of the infrastructure via OMD (Check MK / Nagios)
  • Provided further in-depth monitoring of key aspects of the infrastructure using graphite, with polling intervals as low as 5 sec in most places
  • Developed custom synthetic-transaction-style monitoring checks that accurately simulated what a real user would be doing
  • Managed team of offshore operations engineers in Kolkata, India
  • Implemented log aggregator built on Elasticsearch-Logstash-Kibana stack providing developers real-time access to production logs
  • Integrated New Relic with some key "underperforming" applications, allowing dev team to immediately find and fix performance bugs
  • Automated process of virtual machine provisioning
  • Automated entire process of WordPress web site management (new site, update site, etc.)
  • Developed custom integration with SaaS ticket provider, ServiceNow
  • Developed an Active Directory account unlock / password reset tool
  • Implemented Red Hat Spacewalk for KickStart, Config Management, Local RHN
  • Implemented internal yum repository for all custom packages and internally developed applications
  • Developed and tested disaster recovery plan utilizing NetApp SnapMirror and VMware
  • Automated full install of Oracle 11gR2 via custom puppet module
  • Automated process of Oracle database cloning from NetApp snapshot
  • Automated process of Oracle database hot backup
  • Optimized Linux kernel for Oracle and NFS performance
  • Implemented Oracle Direct NFS for increased performance
  • Ran multiple Oracle databases on NetApp NFS volumes with screamingly fast I/O (10GB/s per db)
  • Optimized NetApp aggregate and volume for Data Warehouse
  • Advanced monitoring of NetApp filer using custom data collectors and graphite, achieving avg 15ms polling intervals
  • Performed routine NetApp hardware maintenance (drive swap, card swap, motherboard swap, etc.)
  • Performed multiple NetApp Data OnTAP non-destructive upgrades with no user impact
  • Developed several custom f5 iRules for a variety of advanced functions
  • Automated several Active Directory management tasks
  • Served as mentor role to junior engineers, fostering growth into more advanced roles
  • Implemented PagerDuty for a consolidated and standardized delivery system of all types of monitoring alarms
  • Participated in weekly pager rotation

Director of Technology

Digi-Net Technologies, Inc. Jan 2006 - Dec 2010 Atlanta, GA

Senior-most technical role in a small software development start up. Managed all software development and infrastructure supporting a varying range of Java-based SaaS applications
  • Managed team of offshore developers in Novosibirsk, Russia, traveling to Russia for one month
  • Managed team of freelance development consultants
  • Implemented agile development methodologies including scrum
  • Led development effort of a visitor profiling and engagement engine, used by Amtrak.com
  • Led development effort of real estate web marketing application
  • Led development effort of social network email client application
  • Led development effort of AJAX chat client, used by team of chat operators supporting a network of online education sites
  • Led development and maintenance of legacy applications
  • Optimized AJAX chat client for improved page load times
  • Architected and implemented a complete private cloud infrastructure utilizing Dell Servers and Storage, and XenServer
  • Migrated all legacy applications and physical servers to private cloud infrastructure
  • Developed custom automated build / release processes
  • Developed a custom reporting and analytics engine using AWS
  • Developed a web based analytics viewer for SaaS customers
  • Developed a custom reporting and analytics system integrated with Sharepoint
  • Developed custom interface to export statistical data from application into Amtrak.com data warehouse
  • Developed custom config management system using subversion + bash
  • Developed custom JMeter tests to simulate the load of thousands of visitors simultaneously
  • Implemented software load balancing solution using httpd / mod_balancer capable of receiving over 5000 requests per second, supporting all visitors to Amtrak.com web site
  • Implemented software load balancing solution for SaaS application using Pound, with HTTPS/SSL/TLS termination
  • Implemented and maintained SMTP delivery system that delivered email blast to a list of one million weekly
  • Developed custom monitoring checks for SaaS application simulating all actions taken by real user
  • Automated / standardized website content publishing using subversion
  • Automated Nagios monitoring via config file generation
  • Served as role of Sales Engineer for potential "big" clients, representing our technology to executives
  • Created custom demo sites branded to match customer sites for purposes of showcasing our technology

Senior Systems Administrator

Digi-Net Technologies, Inc. Apr 2002 - Jan 2006 Gainesville, FL

Systems Engineer embedded into a software development team. Managed all infrastructure supporting a varying range of Java-based SaaS applications as well as all internal IT infrastructure.
  • Managed Cisco router with load balanced T1's for office connectivity and server hosting
  • Automated provisioning process of new SaaS accounts
  • Replaced all internal IT infrastructure (web, dns, file shares, etc.) with Linux (previously mixture of Windows NT, 2000 and Mac OS)
  • Defined and implemented standardized installation and configuration guidelines for all servers in environment
  • Migrated servers from local hosting to managed hosting provider
  • Implemented monitoring and alert solution for environment where none existed previously
  • Provided 24/7 pager support
  • Published content to public web sites
  • Developed and maintained custom PHP forms on public web sites
  • Implemented limited virtualization environment using VMware Server
  • Automated build and release process for all applications
  • Designed, tested, and ultimately successfully executed Disaster Recovery plan during Florida's 2005 hurricane season
  • Served as escalation point for general network and desktop support issues

Product Support Manager

Digi-Net Technologies, Inc. Sep 2000 - Apr 2002 Gainesville, FL

Managed a team of product support engineers, supporting a varying range of Java-based applications on the web. Also provided senior operations support for SaaS application.
  • Developed custom issue tracking / management system for internal use
  • Wrote all of the end-user documentation for all products
  • Served as both technical and customer service escalation point for customers and product support engineers
  • Provided on-site training and professional services support for enterprise-level customers
  • Defined and implemented standardized support procedures
  • Mentored junior engineers, fostering growth into more advanced roles
  • Responsible for all software releases, both customer download and SaaS environments
  • Responsible for all operations of servers hosting SaaS applications

Product Support Engineer

Digi-Net Technologies, Inc. May 2000 - Sep 2000 Gainesville, FL

Provided end-user support for a Java-based chat application for a small software development start up.
  • Provided end-user support to customers via phone, email, and chat
  • Guided customer through installation of Java-based server software on a variety of operating systems
  • Guided customer through integration process of Java applet into their existing web sites
  • Provided basic web development support
  • Quickly excelled to role of team lead

Skills

Build / Release Management
CI/CD, Gradle, git, subversion, rpmbuild, fpm, yum, Jenkins, Jenkins Pipeline, Jenkins Job DSL, Maven, Artifactory, Ant
Infrastructure as Code
CloudFormation, Packer, Vagrant, Puppet, Hiera, AWS Lambda, cfn-init, Cloud-Init
Configuration Management
Puppet, AWS Systems Manager Parameter Store, AWS Systems Manager SSM Agent, Zookeeper, Red Hat Spacewalk/Satellite
Agile
Scrum, Kanban
Networking
AWS VPC, AWS VPN Gateway, AWS Security Groups, iptables
Logging
log4j, Python logging, CloudWatch Logs, syslog, Elasticsearch Logstash Kibana (ELK)
Web Development
HTML, CSS, Javascript, jQuery, Bootstrap, Django
Scripting
Groovy, Python, bash, PHP, PowerShell
Programming
Python, Groovy, Java
Storage / NAS
AWS S3, AWS EBS, AWS EFS, NetApp (Data OnTAP 7,8), NFS, CIFS/SMB, iSCSI
Virtualization
AWS EC2, AWS EC2 Auto Scaling Groups, VirtualBox, VMware vSphere, Citrix XenServer
Containers
Docker, Docker Compose, AWS ECR, AWS ECS, AWS Fargate
Databases
AWS RDS, AWS RDS Aurora, MySQL, PostgreSQL, Oracle, MongoDB, Elasticsearch
Monitoring
CloudWatch, Icinga, New Relic, Nagios, Check MK, OMD, graphite, logstash
Server Platforms
Linux (Amazon Linux, RHEL, CentOS, Debian), Windows Server, ESXi
Application Servers
JBoss, Tomcat, WebLogic, Apache httpd, WSGI
Load Balancers
AWS ELB, AWS ALB, AWS NLB, HAProxy, nginx, f5 BIG-IP
Directory Services
FoxPass, FreeIPA, Active Directory, OpenLDAP
DNS
AWS Route53, PowerDNS, Bind, Active Directory
Automated Testing
JMeter, Selenium, PhantomJS
Applications
Google Suite, Slack, Duo, GitHub, Stash/BitBucket, Jira, Confluence, LastPass, Office365, ServiceNow, PagerDuty