cloudy-chance-of-sun

weekly overview - what’s new?

Welcome to another issue of CloudCast. We do not have aggregate numbers for January’s labor market trends, so we will be doing a deep dive on another role profile (AWS Solution Architects), covering common problems within cloud governance, and talking about some open source tools we think are paving roads within the domain.

breaking down “cloud governance”

Similar to how we organize the roles of cloud professionals by commonly used tools, we organize tools by the problems they were built to solve. In short, by taking a collection of tools within a domain we can generate a list of “problem categories”. Using this strategy on “cloud governance” we find the following:

• inventory: user needs full visibility into resources that are being used
• usage: user needs metadata associated with resource usage  
• pricing: user needs pricing data for usage metrics
• tagging: user needs to apply business logic / labels onto cloud resources
• kpi: user needs to determine health of the system, as defined by the business, at various dimensions
• visualization: user needs to tell a story with the data to generate buy-in
• alerting: based on business logic, user needs to be alerted of anomalies
• policy: user needs to create structural changes to an environment, based on business logic (e.g. disable console access to account A, restrict access to resource type X in account B)  

There are additional niche categories, but these we believe to be the most foundational to a strong cloud governance practice.

a deep dive into demand on solution architects

With 42k open unique roles in the last three years, solution architects are one of the more popular roles, in addition to taking a ~40% premium on their salary compared to the median salary of software engineers.

Considering these two labor groups essentially use the same tools, once we adjust for supply / demand ratios of these groups, we are left to conclude the tools used by solution architects are less capable of solving their specific problems - the same set of foundational cloud governance problems listed above.  

fig 1. demand for solution architects

fig 2. wage distribution for solution architects

fig 3. wage distribution for software engineers

fig 4. wage trends for software engineers, 2010 - present

fig 5. unique posting trend, solution architects

fig 6. top specialized skills, solutions architects

fig 7. top technical skills, solutions architects

Most of the time when we talk about “market share” we are speaking to a company’s share of market’s revenue. This can be misleading. When it comes to a dynamic, ephemeral markets (like the cloud) I like to think of market share in terms of supply and demand for labor and tools.

I.e. What share of companies are asking for Azure solution architects versus AWS solution architects? Answering this is outside of the scope of this post, but something we hope to cover soon.

In this instance, we note, like the FinOps professionals in last week’s issue, there seems to be a deficit in python skill sets, along with DevOps knowledge. Worth noting that, for the first time, we have seen greater supply for solution architects in the US market, than we do demand. I would expect wage pressures for the role to mount as a result.

bringing it all back together

By now we should have a clear understanding of the problem: cloud governance is hard due to poor tooling. We also have a clear understanding of specifically which tasks need to be completed to consider this problem “solved”. Now we can go build something, or find some cool tools that are already tackling these problems. 🙂

Resoto: Creates an inventory of your cloud ☁️, provides deep visibility 👀, and reacts to changes in your infrastructure.

Cloud2SQL: Read infrastructure data from your cloud ☁️ and export it to a SQL database 📋.

Rill Data: Rill makes it easy to create and consume metrics by combining a SQL-based data modeler, real-time database, and metrics dashboard into a single product—a simple alternative to complex BI stacks.

Cloud Custodian: CC enables you to manage your cloud resources by filtering, tagging, and then applying actions to them. The YAML DSL allows definition of rules to enable well-managed cloud infrastructure that's both secure and cost optimized.

Infracost: Cloud cost estimates for Terraform in pull requests💰📉

Infracost pricing api: GraphQL API for cloud pricing. Contains over 3M public prices from AWS, Azure and GCP. Self-updates prices via an automated weekly job.

Put another way…

• inventory: resoto, cloud2sql
• usage: resoto, cloud2sql
• pricing: infracost, resoto, cloud2sql  
• tagging: cloud custodian, resoto
• kpi: ?
• visualization: rill data
• alerting: cloud custodian, infracost
• policy: cloud custodian

conclusion

Like parts to a car, a piece of software serves a specific purpose in your business. To select the right tooling, we suggest first defining the problem space, then comparing features of the technologies within it to get a better idea of the problem’s foundation, then make build versus buy decisions based on your, and your colleague’s, skill sets.  

This post ran on a little longer than expected, so we’ll be saving the walkthroughs where these cloud governance tools have been used in action for the upcoming weeks. Stay tuned.  

resource links

• https://github.com/boto/boto3
• https://github.com/someengineering/resoto
• https://github.com/someengineering/cloud2sql
• https://github.com/rilldata/rill-developer
• https://github.com/cloud-custodian/cloud-custodian
• https://github.com/infracost/infracost
• https://github.com/infracost/cloud-pricing-api

You just read issue #2 of CloudCast. You can also browse the full archives of this newsletter.

‍