[{"content":"We all have errors. What is your nominal error rate? Does that include customer errors or just internal errors? Is everyone on the same page about that?\nHow can we get the highest signal fault attribution data when we\u0026rsquo;re looking at metrics in our APM tools? We can break the challenge down on two dimensions.\nError reporting precision: More granular status classification than HTTP codes get you by themselves, including the ambiguity of fault assignment. This also covers APIs that only use a single HTTP code, like industry-specific XML APIs that report custom status in the body. Error reporting accuracy: Propagation of and voting on fault assignment as requests travel through the depth and breadth of a microservice call graph. Solving both dimensions reaps several benefits.\nStraightforward, readable monitoring queries. Fault assignment is already pre-computed, avoiding the need for brittle queries. Query conditions stay bounded as system size and failure modes grow. Business-approved classification definitions lead to understandable SLOs that resonate. Your choice of rough or fine-grain status buckets is available depending on the monitoring or reporting use case. Use rough grain for more anomaly detection statistical power or fine grain for a jump start on troubleshooting. Classification Classifying errors in deeper granularity increases precision. Sometimes you don\u0026rsquo;t want precision, like when you need more statistical power in anomaly detection monitoring. In those cases, the sub-classes can be combined back together into higher level classifications: success, our fault, customer\u0026rsquo;s fault. Maintaining more precise, per-request outcomes is helpful for troubleshooting or assessing incident impact.\nHere was a quote from David Reaver in his talk How Stripe achieves five and a half 9s of availability, Re:Invent 2024 that is a useful data point how some organizations are moving beyond just using status codes. I would be curious to learn more about Stripe\u0026rsquo;s error classification framework.\nWe don\u0026rsquo;t solely rely on standard HTTP response codes to decide if a request is successful or not. Classically 200 is \u0026ldquo;that\u0026rsquo;s great,\u0026rdquo; 500 is \u0026ldquo;error and our fault,\u0026rdquo; 400 is \u0026ldquo;error and your fault.\u0026rdquo;\nThat\u0026rsquo;s too simple. We don\u0026rsquo;t do that. There are lots of legitimate 400 responses that are kind of routine and expected. You can\u0026rsquo;t just page on them all the time.\nExamples: 404s on a system of record might have a different level of confidence than 404s on a derived data system. It would be nice if we could engineer all systems to be like KMS grant tokens and overcome eventual consistency, but in reality, sometimes we have to deal with non-ideal scenarios. It can be helpful to recognize and codify a particular system\u0026rsquo;s resource not found as an unclear attribution error instead of just assuming it\u0026rsquo;s the client\u0026rsquo;s fault. We can follow up with automation that checks if the item eventually shows up or correlate with processing/replcia latency metrics coming out of that system to get the full picture and raise the necessary alerts.\n400 errors can also have a large semantic range. One client might send a request with North Carolina instead of NC for an enum value and be rejected immediately. Another might fail further down the request path if North Carolina doesn\u0026rsquo;t yet have regulatory approval for a product. Both failures are based on the customer-provided data. We can classify them more precisely into customer data validation error and customer data error.\nThings can get tricky on classifying faults with conditionally required data. A business rule upstream could necessitate an additional piece of data only sent in 1% of circumstances. Is this a validation error or a calculation error? This is why, irrespective of which classifications your business chooses, they must be defined precisely. The classifications provided here are a starting point to jog ideas. Each organization should agree on a set that covers all their incoming transactions. The organization can then have a shared nomenclature, leading to more accurate communication.\nVoting The second goal of this proposal is to increase error reporting accuracy via delegating and voting. Whether you have microservices, a modular monolith, or any other design, there will be abstractions. These abstractions shield complexity but also shield information. Ultimately, we want to know the status of the final response to the customer, but the high abstraction component last in line on the response path might not know the details of why a failure occurred. The solution is a structured and predictable way to pass and merge status from all services in the call graph.\nErrors can occur at any level of the call graph and before, during, or after work.\nBefore: a request comes in, validation fails, and immediately returns. During: an upstream service or DB is unavailable. After: multiple DB or RPC calls are made, then an error occurs merging the results. We need a voting algorithm to merge all reported statuses into a final status. That final status should favor severity first and specificity second. We can visualize this as settling into local minimums on a response surface plot.\nCustomer errors are higher severity than successes. Internal errors are higher severity than customer errors. Customer errors are more specific than unclear attribution errors. Customer data validation errors are more specific than generic customer data errors. This data flow diagram shows how state is updated inside each service as well as passed between services. I only show 5 services and a database, but this could scale to very large call graphs as the data added to the response remains a constant size.\nThe data flow looks similar to distributed tracing, but there are a couple of important differences. At scale, distributed tracing samples only a small percentage of requests, while this status reporting mechanism runs on every single request. Distributed tracing also passes IDs between services and then uploads trace data to a monitoring service by ID. This design just passes a value back in response headers or in a metadata section of the response body. Those two differences make this cost-effective to run on every request.\nFeedback What do you think? Should we just attach very explicit business meaning to HTTP codes? Does leaving them decoupled retain more flexibility for evolving classifications while keeping the system API stable? Are there existing standards or frameworks I can research that accomplish a similar objective?\n","permalink":"https://blog.danieladamstech.com/2024/transaction-status-voting-proposal/","summary":"We all have errors. What is your nominal error rate? Does that include customer errors or just internal errors? Is everyone on the same page about that?\nHow can we get the highest signal fault attribution data when we\u0026rsquo;re looking at metrics in our APM tools? We can break the challenge down on two dimensions.\nError reporting precision: More granular status classification than HTTP codes get you by themselves, including the ambiguity of fault assignment.","title":"Transaction Status Voting Proposal"},{"content":"Today, we\u0026rsquo;ll look at two over-the-air transmission use cases that each spawned three different protocol standards. I\u0026rsquo;ll include some pattern observations concerning the two sets.\nFirst, the obligatory XKCD.\nThe two domains we\u0026rsquo;ll be analyzing are Ham radio digital voice and first-person-view drone video feeds. Both of these existed in analog form, and then digital alternatives came out and started gaining adoption.\nHam Radio Digital Voice Digital Mobile Radio (DMR) D-STAR Yaesu System Fusion (YSF) Analog predecessor: Frequency Modulation (FM) D-STAR was the first of the main three digital voice modes by a long shot, starting development back in the 1990s. Then YSF launched in 2013, and DMR came from the commercial space to amateur space in 2014. YSF sought to improve on the status quo in the area of compatibility with analog FM through automatic mode detection and switching. It was never truly published as an open specification, which gave some people reservations about adopting it. DMR published its first set of specs (tier 1 and tier 2) in 2005 and tier 3 in 2012. Commercial radio manufacturers adopted the standard and began making products. Despite the standard being \u0026ldquo;commercial-first\u0026rdquo; and implementations less friendly for amateur use, the on-air protocol had advantages. The main one was that it uses time-division multiple access to multiplex two conversations on the same frequency channel. Simultaneously transmitting multiple concurrent data streams is something we are used to now in packet and IP-based systems we have today, but historically, radio voice transmission had been a single stream per frequency.\nFirst-Person-View Video Transmitters DJI O3 HDZero Walksnail Analog predecessor: NTSC or PAL, like analog TV Drone runtime software video transmitter documentation gives a concise comparison between the three standards.\nHDZero is uncompressed, so all pixels are sent independently. This allows transmission errors to be isolated to single pixels instead of being more intertwined with larger sections of the image as is common in video compression codecs. Transmission is one-way with no retransmissions, similar to UDP. Information that does arrive has the lowest possible latency, and we just deal with missing information. The main use case for HDZero is in racing or other shorter-distance situations where the lowest latency is required to enable quick reaction times.\nDJI and Walksnail follow more of a TCP approach. The receiver will ask for retransmission of any data that was missed. It usually succeeds on the second or third attempt, so in the end, the pilot has a perfect picture. As the signal degrades, it isn\u0026rsquo;t the quality of the video feed that goes down, it\u0026rsquo;s the latency as more transmission attempts are required. The retransmissions allow these systems to have greater overall range at the expense of higher latency than single-shot systems like analog and HDZero.\nObservations There will always be multiple standards in anything that has utility or economic opportunity. That is the great thing about competition in the free market. The existence of other standards pushes everyone to iterate and improve their offering. After a couple of years, things settle down into a couple of winners. Generally, these will have different tradeoffs that will appeal to different use cases. Once the major use cases are covered, there\u0026rsquo;s not as much of a reason to create a new standard as the incremental improvement would be small and it would be an uphill battle to displace existing market share.\nA pattern I see in these two situations (potentially overfitting on this small data set) is a set of three top runners with two approaches. Two of the standards use a similar approach, and the third has something fundamentally different. In the digital voice category, DMR is the standout with its timeslots capability. HDZero is by itself in drone digital video with its one-way transmissions.\nBeing different does not necessarily make you the winner. DMR has \u0026ldquo;won\u0026rdquo; in the last couple of years as the favorite (depending on who you ask.) Their ability to have multiple concurrent conversations on different time slots is the killer feature that sets them apart. HDZero makes tradeoffs with downsides (lower latency for less range.) This makes it a use-case-specific decision.\nUltimately, winners must manage additional factors than just what provides the best capability for the user (VHS vs Betamax). Despite being the favorite, I am predicting DMR will take 10 years before it has a majority of the market share. Communication protocols have a strong network effect. You want to be on the same one as your friends. If a metropolitan area already has several repeaters (radio towers) of a certain variety, the next incremental installation will probably be more of the same. Yaesu flooded the market with at-cost repeaters around 2018-2021 to establish themselves as the dominant protocol. They knew they would make the money back on the handheld radios people would buy to connect to the repeaters.\nThese are just two sets of standards I\u0026rsquo;ve come across lately in hobby pursuits that seemed to parallel. I\u0026rsquo;m curious what other areas might be out there that exhibit similar phenomenon.\n","permalink":"https://blog.danieladamstech.com/2024/three-competing-standards/","summary":"Today, we\u0026rsquo;ll look at two over-the-air transmission use cases that each spawned three different protocol standards. I\u0026rsquo;ll include some pattern observations concerning the two sets.\nFirst, the obligatory XKCD.\nThe two domains we\u0026rsquo;ll be analyzing are Ham radio digital voice and first-person-view drone video feeds. Both of these existed in analog form, and then digital alternatives came out and started gaining adoption.\nHam Radio Digital Voice Digital Mobile Radio (DMR) D-STAR Yaesu System Fusion (YSF) Analog predecessor: Frequency Modulation (FM) D-STAR was the first of the main three digital voice modes by a long shot, starting development back in the 1990s.","title":"Three Competing Standards"},{"content":"I\u0026rsquo;ve been following Will Larson for a while at this point. I read his book Staff Engineer, listened through the associated podcast, , follow his blog, and listened to his talk on strategy. I was excited to see his interview pop up on Lenny\u0026rsquo;s podcast recently.\nThen, I heard Lenny\u0026rsquo;s interview with Richard Rumelt and thought Richard had a straightforward, practical approach to thinking about strategy. It\u0026rsquo;s interesting tracing the lineage of ideas - Richard Rumelt\u0026rsquo;s mental model was similar to what I had heard from Will Larson. It makes sense; Will had read and drawn on two of Richard\u0026rsquo;s books. He has blog posts Notes from \u0026ldquo;Good Strategy, Bad Strategy\u0026rdquo; and Notes on The Crux. I\u0026rsquo;m going to keep this post short since I don\u0026rsquo;t have much of anything to add to the discussion. These are just the highlights that most stood out to me.\nStrategy has 3 components: Accurate and honest diagnosis Guiding policy Coherent and consistent action Listening to the full podcast will give the fullest explanations and anti-patterns of the three elements. I will highlight one anti-pattern here. That is equating goals with a strategy; even worse, a grab-bag of unfocused goals. Goals have the potential to be more nebulously attached to both the input (diagnosis) and output (coherent actions.) A guiding policy should be strongly, causally attached to the other two components. The connection between the first two ensures you are solving the right thing, and the second tie contributes to the efficacy of your solution. A clarifying, alternative term for strategy is \u0026ldquo;Action Agenda\u0026rdquo; (aligning with Amazon\u0026rsquo;s leadership principles, Bias for Action.) \u0026ldquo;Action Agenda\u0026rdquo; emphasizes that the only useful strategy is one that moves people to action. Without action, nothing will be accomplished.\nWhen contemplating a new project or finding yourself in a competitive position, tackle the crux. The Crux is the hardest part of a rock climb. Richard drew the parallel that if you can\u0026rsquo;t handle the crux of a climb, you shouldn\u0026rsquo;t do the climb at all. Similarly, not being able to tackle the crux of a product or business problem means you should probably table the idea. You don\u0026rsquo;t want to waste effort on other portions of the sequence if you can\u0026rsquo;t make it to completion. That isn\u0026rsquo;t to say we should just be giving up. An encouraging piece of advice was to \u0026ldquo;Pick a problem that aligns with your ambition but is also reasonable to do something about.\u0026rdquo; Finding that most important problem where you can move the needle and taking consistent action will almost always lead to improvement.\nAn understanding of power and history are two contexts key to creating a strategy. Power is an angle differentiating you from the competition that you can exploit as an asymmetry. Network effects and distribution channels are two examples of power. Richard put a big emphasis on reading business, military, and economic histories as a way of understanding both strategies that worked and ones that failed. He says to be a mentally active participant in the historical situation, taking in the information people had at the time and analyzing it. One way to do that is by reading the front page of old newspapers like the New York Times. That helps immerse yourself in the context of the moment and set aside biases of 20/20 hindsight we have when looking back on the situation.\nIn conclusion, I would highly recommend a listen for anyone wanting to level up their strategy analysis game. It is a hard choice whether I should add Good Strategy, Bad Strategy or a history book to my reading list! I\u0026rsquo;m almost done with Never Split the Difference and have Rob Henderson\u0026rsquo;s book Troubled coming up on family stability and social class.\n","permalink":"https://blog.danieladamstech.com/2024/strategy-with-richard-rumelt-and-will-larson/","summary":"I\u0026rsquo;ve been following Will Larson for a while at this point. I read his book Staff Engineer, listened through the associated podcast, , follow his blog, and listened to his talk on strategy. I was excited to see his interview pop up on Lenny\u0026rsquo;s podcast recently.\nThen, I heard Lenny\u0026rsquo;s interview with Richard Rumelt and thought Richard had a straightforward, practical approach to thinking about strategy. It\u0026rsquo;s interesting tracing the lineage of ideas - Richard Rumelt\u0026rsquo;s mental model was similar to what I had heard from Will Larson.","title":"Strategy With Richard Rumelt and Will Larson"},{"content":"In my time off between Christmas and New Year, I generally read a book. Two great non-tech books I\u0026rsquo;ve read in past years are Range: Why Generalists Triumph in a Specialized World and The Spy and the Traitor: The Greatest Espionage Story of the Cold War on Oleg Gordievsky. Both of those were gifts from my brother.\nThis year\u0026rsquo;s book was The Software Engineer\u0026rsquo;s Guidebook by Gergely Orosz. I am a reader of his newsletter, The Pragmatic Engineer. The book traces through all software engineering career levels and highlights important practices to be following at each. I wanted to read through all of them and catch up on any tips or behaviors I had missed along the way. Below are groupings of advice I found helpful.\nOwn Your Own Career YOU are the one responsible for your own continued learning and growth in your career! Reflecting on my career growth, I have gone hard in most areas. These include broad and deep skill building as well as certifications. One activity I regret not starting sooner was signing up as an official mentee in the mentor program. I had worked on the team with our two Enterprise Architects for 3 years and learned a ton from them in an informal mentee relationship. I had some impostor syndrome if I was truly ready for a mentor from the business or would run out of things to ask them halfway through the year and look dumb. I\u0026rsquo;m so glad I took the plunge! Both of my mentors have been very friendly and gracious. I switched partway through last year since my first mentor took a position as VP of Sales with around 140 reports at another company. Another career development move I know I need to make at some point is working for another company to get experience in scale. That one I\u0026rsquo;m going to need to time what works for my family, potentially the economic cycle/ ZIRP, and navigate some more impostor syndrome.\nIn the eight years since college, I have been on six teams. Two of those I started from scratch (Cloud Engineering and Cloud Data Engineering.) I expanded my current Integrations team from being a business analyst group that borrowed developers from the general Services team to being a self-contained product team with a software engineering arm. I feel like I\u0026rsquo;ve stayed a good balance of time in each domain to grow deeper without the learning rate petering off. I currently hold six AWS Certifications: Architect and DevOps Pro, Data and Security Specialties, and the associate certs. My lapsed certs include CompTIA A+ (high school), Security+ (college), and Red Hat Certified Systems Administrator and Engineer on RHEL 7 (after college.)\nThe current balance I am trying to strike is how to best spend my time between coding and non-coding activities. I probably code between 30 and 40% of my time. I want to spend the remainder in high-leverage direction-setting, doc writing, and mentoring tasks. In the project management realm, I am trying to reflect on what useful ways I can contribute to coordinating smooth delivery while at the same time reserving time to personally help with some of the more critical migrations and implementations.\nBe Known as Someone Who Delivers A prerequisite for being known as someone who delivers is to actually deliver. You should first be consistent, dependable, and meet or exceed the quality bar (see later about quality not being the end-all-be-all in some situations.) Then you can worry about whether others are recognizing your results. It is important to not put the cart before the horse and try to sell something that\u0026rsquo;s not there. Even when you are solid on the behavior, selling it can require walking a tightrope. You need to get a feel for what is acceptable in the engineering culture and not deviate too far from the norm. You don\u0026rsquo;t want to come off as someone who just cares about their image more than the actual business outcomes. On the flip side, you don\u0026rsquo;t want to sit on the sidelines assuming your work will speak for yourself in a company where there is an expectation to self-promote.\nAvoid spreading yourself too thin to deliver meaningful work. My biggest takeaway from the book was \u0026ldquo;know your #1 priority and always deliver that on time.\u0026rdquo; This priority should be stable for at least a week. Within the week, switching to a new #1 priority should require managerial escalation and approval. I thought I was going to adopt this and revolutionize my focus\u0026hellip; Turns out, it still doesn\u0026rsquo;t mean you can just work on one or two things at a time. We are currently working on a major upgrade and cloud migration for one of our core business systems. There were three things we identified the Integrations team needed to contribute for customer rollout over six weeks. Those were data synchronization from an on-prem system of record to DynamoDB, orchestration router service updates, and unifying our customer support tooling. The first two items were in the request path and had a dependency, so I took them. The third was independent, so I was hoping to get the request-path items done first and then start guiding someone on the support tooling. The good news was an offshore contractor had availability sooner than expected, so I started breaking down the work for him right away. At the end of the day, the timeline is still on track working on all three in parallel, just with a bit more context switching.\nBe Flexible and Adaptable Flexibility and Adaptability are critical skills for delivering what is needed in the way it is needed. A company might be in a phase where speed is valued over quality - finding product market fit or launching a feature to catch up with a competitor that is taking your business. Or the company might have reached a point where stability and earning customers\u0026rsquo; trust is paramount. The interesting thing is that the progression between those extremes is not a monotonically increasing function. Some forces can cause oscillation between the two. Gergely cites the example of a startup going into a funding round vs one that just completed a funding round. The book includes an insightful comparison table between peacetime and wartime behaviors and leadership styles along a dozen different dimensions. For example, conflict is acceptable in a wartime company as long as it gets things done. Not so much in a peacetime company - relationships are the higher priority.\nAs I will mention later in the Architect Archetypes section, sometimes the company\u0026rsquo;s needs don\u0026rsquo;t exactly align with our default personality or behavior. That\u0026rsquo;s ok. As software engineers, we should be comfortable in a relatively wide range of modes. Impedance matching power transfer is an interesting parallel to an engineer\u0026rsquo;s fit with company needs. In ham radio, we use radios and coax cable with 50 ohm impedances. The goal is to get the antenna\u0026rsquo;s impedance as close to 50 ohms as possible to maximize power transfer. Power that is not transferred from the cable into the antenna is reflected back towards the radio and raises the standing wave ratio or SWR. We use antennas designed to match 50 ohms as closely as possible or transformers to lower the impedance to the target. A software engineer will be able to do their best work when companies needs align within the range of what they are most comfortable producing. The broader the acceptable range, the more options you have.\nAnother type of flexibility engineers can develop the ability to thrive in either a product or platform team. Both teams build for their customers, but the types of inputs they get vary widely. Product teams have metrics that are easy to align with the business (new users, churn, revenue) while platform teams have an additional layer of indirection (engineering time saved, cost savings against a moving target.) At large companies, platform teams have the highest leverage since they can improve cross-cutting concerns for the whole organization. The best thing you can do as an engineer is to contribute in both team types. I spent more time on the platform side, so I wanted to switch over to a product team. Changing it up gave me a new energy for building a different type of deliverable.\nLeverage Through Writing Informative, succinct, and persuasive writing is key to scaling your influence. You can reach more people by writing than in 1:1 conversations. Writing also memorializes the information for others a year down the road when you have moved on to the next project. We all think of system documentation as the main required form of writing for engineers. But other critical documents exist like RFCs/change proposals, roadmap planning, and \u0026ldquo;north star\u0026rdquo; team vision documents. Contributing to documents influencing what will be done is a high-leverage activity.\nThe quality of the writing you create is more critical in some cases than others. You should be aware of the intended but also potential audience and put in the commensurate level of tweaking. Ryan Peterman, staff engineer at Instagram, says he spends 80% of his writing time polishing the final draft. A cool story on Lenny\u0026rsquo;s Podcast with Ethan Evans was how he wrote the tagline for Ownership when it was on the chopping block. \u0026ldquo;An Owner never says that\u0026rsquo;s not my job.\u0026rdquo; Ownership is currently listed as #2 out of Amazon\u0026rsquo;s 16 Leadership Principles. On the flip side, take time to digest what has been written previously. A great internal starting place the book calls out is the corpus of past postmortem reviews. They can give you a sense of what challenges the company has faced in the past and what improvements or mitigations should be implemented going forward. A good external source of knowledge is The Amazon Builders\u0026rsquo; Library. One of my favorites is Reliability, constant work, and a good cup of coffee about how control planes and other critical systems should do the same amount of work each time and sometimes avoid optimizations. Taking a shortcut in a NOOP condition that occurs when everything is normal and then doing work in edge cases or failure scenarios can overwhelm the system and lead to worse cascading failures.\nProject management One of the ground rules in project management is to build the right thing. Engineers should work closely enough with the business that they can be a trusted extension of the business\u0026rsquo;s decision-making. Alignment and knowing how your business counterpart would respond in various situations allows you to inform and move ahead rather than having to ask and wait for feedback. Stakeholder management is an important skill to be successful as a tech lead. You must ensure all stakeholders feel their input was taken seriously and all parties can arrive at a shared consensus on the project\u0026rsquo;s goals and requirements. During execution, you need to keep them updated on progress, decision points, and risks. I generally provide a stakeholder update at the end of the week of what progress we have made and what is the direction for the upcoming week.\nEstimations are always a challenge, but the vast majority of businesses need them. Sometimes multiple engineering teams need to roll out a joint effort around the same date. Other times Marketing will need to coordinate publicity around a product launch. The estimate always will factor into the decision of whether something is worth doing now, later, or never. The book coins the term \u0026ldquo;Triangle of Software Physics\u0026rdquo; consisting of the variables People, Scope, and Timeline. Just like the ideal gas law, when one variable changes, one or more others must counter to maintain equilibrium. The most common change of which everyone is familiar is increasing Scope! The only solution to that is to increase the People or extend the Timeline. The Mythical Man-Month states that simply adding new people to a project that is behind is unlikely to speed it up. Thus many times people end up working overtime or pushing out other tasks to focus more on the project at hand.\nThe strategy around how a project is broken down and estimated reminds me of the blog post Stepping Stones, not Milestones. Creating waypoints along the path achieves the tandem goals of incrementally delivering value and gaining a clearer picture over time of the end goal. Incremental value still gives you something to show for your efforts if the business priority changes halfway through and the project gets dropped. Also, in large-scale projects, it is hard to pinpoint exactly how things should look at the very end. Starting with a broad end goal and continually narrowing the design over time will help you converge on a good solution.\nBuilding Reliable Software Software reliability is a passion of mine. I don\u0026rsquo;t know that my interest is full-bore SRE, but it skews towards that side. The book covers both the traditional monitoring and alerting topics as well as how companies safely ship to production. These practices on shipping to production are broken down between startups, traditional companies, larger tech companies, and Facebook. It was insightful to compare the various patterns along the dimensions of speed (latency and time spent) and safety. Facebook and Amazon have two different release strategies despite being at the same maturity due to the differing product risk tolerance. A bad update to Amazon RDS databases is a completely different thing than a bad update to Facebook\u0026rsquo;s trending algorithm. After CI checks and code review, Facebook launches an automated canary rollout across 4 successive environments (automated test, employees, small market, all users.)\nThis week I gave a lunch-and-learn on Monitoring and Observability. I highlighted some of the improvements I had worked on for our team over the past quarter. They included starting a NewRelic Terraform project for our team that managed alert conditions, alert routing, dashboards, and synthetics. I\u0026rsquo;m a big fan of infrastructure as code, and this was a step to get consistency across our environments. I also gathered feedback on an RFC I led regarding enterprise-wide baseline monitoring expectations for our services.\nArchitect Archetypes Gergely\u0026rsquo;s taxonomy of software architect traits helped me introspect and see where I fall among these characteristics. Picking which angle to adopt still requires business need awareness. While most people will have a personality tendency towards a subset of these traits, we need to honestly evaluate the business needs and be willing to adjust from our position of comfort to better serve the business. The new-shiny chaser and the old-schooler are the obvious dichotomy. But it is potentially hard to be both a coding machine and leave enough time for being the approachable one. Those are two I try to balance.\nMore Theoretical The ivory tower architect Painfully precise The theory addict The philosopher The superior linguist The walk-away advisor More Practical The coding machine The integrator The approachable one The detailed documenter The new, shiny chaser The old-schooler Wrap Up I would recommend this book for anyone in the first ten years of their software engineering career or for those wanting insight into what it means to be a \u0026ldquo;good developer\u0026rdquo; in big tech. It covers all the need-to-knows, allowing you to feel comfortable that you\u0026rsquo;re not missing anything important in your mental model of how high-maturity software engineers should function. If you know me IRL, I would be happy to loan out this book. I currently have Team Topologies loaned out to a business-side counterpart I worked with when I was on the cloud data engineering team.\nI am about to slide off my fence-straddling position into a paid subscription to Gergely\u0026rsquo;s The Pragmatic Engineer newsletter.:)The problem is a couple of months ago I went on a Substack free subscription spree and now need to figure which others to cut to make time!\n","permalink":"https://blog.danieladamstech.com/2024/the-software-engineers-guidebook/","summary":"In my time off between Christmas and New Year, I generally read a book. Two great non-tech books I\u0026rsquo;ve read in past years are Range: Why Generalists Triumph in a Specialized World and The Spy and the Traitor: The Greatest Espionage Story of the Cold War on Oleg Gordievsky. Both of those were gifts from my brother.\nThis year\u0026rsquo;s book was The Software Engineer\u0026rsquo;s Guidebook by Gergely Orosz. I am a reader of his newsletter, The Pragmatic Engineer.","title":"The Software Engineer's Guidebook"},{"content":"My sister got me the Grecian Computer Brainteaser Puzzle for Christmas. My first thought was: \u0026ldquo;Should I write a program to brute force it or go the cryptanalysis route to find a weakness?\u0026rdquo; After a bit of playing, I didn\u0026rsquo;t expect to have the time I would need to look into the positional relationships to the level necessary for a manual solution. I then pivoted to think about how I would solve it programmatically.\nHere is my Python notebook solution. My math professor used to have a saying about the \u0026ldquo;brute force and sheer ignorance\u0026rdquo; approach. Unless the computational complexity proved too much, that was going to be my first angle of attack. I would run through each of the possible shifts, add up the numbers in each column, and print out if there was a match.\nI followed these steps:\nDraw up a small 4x4 sample matrix solution on paper. Decompose it into 5 layers (also on paper) since the real puzzle had one more layer than rows. Enter these rows into 2-dimensional sparse arrays with None for when you should \u0026ldquo;look through\u0026rdquo; to the next layer down. Write the algorithm. Run it, expecting to find a solution on the first attempt since it was already aligned. Shift the columns around, expecting to still find one solution partway through the brute force search. Write the code to read the matrix definitions from a set of CSV files. Verify CSV ingest method by entering my shifted 4x4 matrices. Enter the real set of five 4x12 matrices. Run the search and verify the solution! I did not know if I would succeed on the first try, so I kept the code unoptimized to allow for the easiest debugging possible. For example, I materialized a list of shift combinations instead of using divmod() to calculate each one on the fly. I kept the actual sums as a list rather than just counting the number of target value matches. I expected to have a typo somewhere in my CSV files and only get 10 or 11 matches out of the full 12. If my optimistic run of the program didn\u0026rsquo;t yield a solution, I planned on backing down the threshold to print state until I got some results. Then I could align the physical dial and check if I entered a value incorrectly.\nOne interesting finding on the first pass was the number of solutions equaled the number of columns with all the shifts incrementing together. This was due to the fact that rotating the full puzzle a slot is still solved. I realized this during my small test run and adjusted the shift generation code to \u0026ldquo;hold one disk steady\u0026rdquo; while rotating the remaining 4.\nI was surprised at how quickly the code ran considering it was O(n^7) == height ^ width ^ depth ^ 4 shifts. I was also surprised to find the solution without having to add any additional debugging! It was a very fun experience solving the puzzle. It ended up taking me 3 hours with much less frustration than if I had tried doing it manually! Thanks again to my sister for a great Christmas present idea!\n","permalink":"https://blog.danieladamstech.com/2024/grecian-computer-puzzle/","summary":"My sister got me the Grecian Computer Brainteaser Puzzle for Christmas. My first thought was: \u0026ldquo;Should I write a program to brute force it or go the cryptanalysis route to find a weakness?\u0026rdquo; After a bit of playing, I didn\u0026rsquo;t expect to have the time I would need to look into the positional relationships to the level necessary for a manual solution. I then pivoted to think about how I would solve it programmatically.","title":"Grecian Computer Puzzle"},{"content":"I was thinking about how to best chronicle my career so far showcasing the variety of contributions I have delivered. After pondering, I came up with the idea of a graph showing my focus between the areas of software, data, cloud, and security.\nI had the look I wanted in mind but wasn\u0026rsquo;t sure how to implement it. My first concept was a stacked bar graph with variable width bars depending on my time in the position. Stacked bar graphs were easy, I did not find many resources on variable width bars. The Python Graph Gallery led me to realize a stacked area chart would be a natural fit. By increasing the number of points, the shape would asymptotically approach a variable-width stacked bar graph. Using the number of months as the width gave me a visually pleasing slope angle at the transition points and was logical as well! Removing the percentile ticks on the Y axis and customizing label positioning on the X completed the graph.\nSee the Python notebook and JSON definition I used to make this graph.\nMy Tech Journey This is some context you won\u0026rsquo;t find on my Linkedin profile. My first introduction to programming came when I took AP Computer Science in 11th grade. Ever since then, I was hooked. When looking at colleges and scholarships, I came across a program with the NSA that would have been a scholarship, summer internships, and a job after college. I made it to the last round at Fort Meade including full-scope poly for TS/SCI clearance but didn\u0026rsquo;t get the offer. I still liked the cyber security angle. During college, I competed on our cyber defense team. Our team placed 2nd in the 2014 Mid-Atlantic Collegiate Cyber Defense Competition. During college, I did a 8-month full-time co-op with Babcock and Wilcox mPower on their small modular reactor building UI prototypes for control room operators. There were a variety of reasons the project did not make it over the finish line, including economic timing and regulatory hurdles, so the next summer I moved on to Genworth with an endpoint security internship.\nAfter college, I started the IT Leadership Development Program at Genworth. This program included four 6-month rotations on different teams to broaden my horizons and help pick the area that was the best fit for roll-off. I started out in Richmond working on Box and OneDrive data loss prevention. Then I worked on the ETL team where I took fraud records from Hive and sent them to an ESB to ingest the data back into the transactional system and surface to underwriters. During this time, Atul Saurav was a great mentor teaching me a lot about data. Still thinking I wanted to stay a security guy, I moved down to Raleigh for an application security rotation on the Mortgage Insurance business unit\u0026rsquo;s security team. I\u0026rsquo;ve been here in Raleigh ever since! The cloud was a hot new topic at that time. After hearing there was a team working on AWS that had recently launched a new EMR-based analytics platform, I worked towards getting on that team. My thought was that this would be an opportunity to still keep a security lens but move earlier in the design process. I created a git repo with terraform labs that I worked on in the evenings to show I was serious about learning AWS. This was around the time AWS ALBs were coming out and the company was migrating off a Consul-based stack to native ALBs. So in my personal AWS account, I built a two-tier alb system for public and private services. When I presented my initiative to Chris, at that time Director of Architecture and now CTO, he gave me the opportunity to rotate on his team. At one point in the six month rotation as I was working through the various projects, he said, \u0026ldquo;As long as you keep hitting it out of the park, I\u0026rsquo;ll keep giving you more stuff to try.\u0026rdquo;\nRoll-off time came, and I received an offer to stay on the Enterprise Architecture team. I was very thankful for the opportunity! Mark and Mike, the most senior individual contributors who had worked as a team of two for the previous 10 years, let me join them. It was a great learning opportunity to be around them during the early years of building out our cloud environment. Guiding principles were codification of everything, strong environment parity, and cross-environment/business context isolation. That team functioned as the cloud center of excellence and delivered big projects partnering with the various development teams. As the obviously junior member, my projects were more internal-facing platform elements, but there were some that started working on my influence skills across teams. One of those was leading the migration from DataDog to NewRelic (back when they had the strongest APM/distributed tracing offering) and driving adoption across all our development teams. We rolled it out successfully, and years later I am still the go-to SME on performance monitoring. Currently, I lead a cross-team monitoring focus group sharing best practices and helping others level up our monitoring across all our apps. Another project of interest in my early years was working with other teams (including HQ approvals) to build out our business unit\u0026rsquo;s own ADFS cluster primarily for federating to AWS. This was years before IAM Identity Center, so I built a utility site allowing us to use SAML roles for CLI and SDK access. The site was server-side rendered and built on Lambda and API Gateway. It parsed the SAML assertion and presented a similar screen to the main AWS sign-in, though with Bootstrap styling. Users could click a role and the AWS JS browser SDK would call STS for temporary credentials. These were rendered into a cross-platform set of aws configure commands that could easily be pasted into a terminal to refresh the appropriate profile. I also worked on a proposed enterprise-wide API fronting a tokenization provider. Circumstances changed and that project was implemented at another time with another solution, but it was thinking through the use cases and writing that facade service. The Architecture team was the position I stayed in the longest at 3 years, trying to learn by doing and osmosis from the experienced people around me.\nOur cloud footprint kept growing and reached the point where we needed a dedicated team to manage it. I was selected to bootstrap the new team. We started out as just the cloud team, but around that time (early 2021) also started building a cloud data engineering team. At the end of my time in the position, we had grown to a team size of 9 FTEs and 6 contractors. That included a Director of Cloud+Data and a Data Engineering manager. There have been jokes made about the number of people needed to replace me as my switching teams was what spurred Cloud Engineering to get their own manager and further expand the team.:) Throughout the whole growth phase for both cloud and data, I was in the technical interview loop for every individual contributor candidate. In September of 2021, we announced our IPO on Nasdaq as Enact (ACT). I had my moment of fame with my youngest in Times Square!\nBesides the organizational, cross-training, and coordinating angle, I also dove into the technical opportunities. We follow a strict infrastructure as code practice where all AWS resources are in Terraform. This includes both our ECS/Lambda services and infrastructure pieces. I made improvements to our workflow in both cases. I reduced 24 different Terraform ECS service deploy pipeline templates into 1 \u0026ldquo;standard\u0026rdquo; service type and a python-generated multi-region version. On the infrastructure project side, I stripped our wrapper scripts to the bare minimum while increasing variable derivation flexibility and reducing ongoing maintenance of script customization. My time on this team was nicely split into two 18-month stints of cloud focus and data focus. I was heavily involved in the design and buildout of our Snowflake data lake. We used Talend for data integration, orchestrating the ELT process, and tracking SQL-based lineage. I built out the development processes, repo structure, CI/CD, Snowflake environment, Snowflake security model, Talend AWS infrastructure, Eventbridge patterns, MLOps pipelines, documentation, and assisted with ETL design where appropriate. One of the big wins on the CI/CD front was adding SQL linting and static analysis with SQLFluff. That addition to the pipeline saved hours of data engineer time by speeding up cycle time to find certain errors instead of waiting for jobs to run. I had the opportunity to collaborate closely with the data scientists by being the interface between the two teams. I worked with the Data Science leads to develop their processes, create their sandbox environment, and help with last-mile model productionalization.\nAt the start of 2024, I felt the need to get back into software. There was plenty of scripting and software-adjacent coding going on, but I was missing the all-in experience. I transitioned from a hybrid Architecture team and a Platform Cloud+Data team to a true Product team when I joined Integrations. We are responsible for our external customer interactions. We own tracking and normalization layers and then hand off to other teams who do rate quotes, orders, and backoffice servicing functions. Today 80% of our business comes through integration channels managed by our team. Collaboration between all the teams is essential for a good customer experience as each one tackles a different angle of complexity. The integration team\u0026rsquo;s mission is to maintain a consistent API for all our 1,800 B2B customers. To that end, we participate in the MISMO standards organization, similar to the IETF for the web. MISMO focuses on standardizing business process touch points and data interchange formats between all parties in the mortgage process. Our corporate values are Excellence, Improvement, and Connection. As the face of Enact technology to our customers, we want to be customer-obsessed, just like Amazon\u0026rsquo;s first Leadership Principle. Customer Obsession\nLeaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.\nIt has been a great time so far building things that I can see our external and internal customers use, whether that\u0026rsquo;s launching a new MISMO 3.5 Rate Quote API or a new query tool to provide support a unified view of our on-prem and native cloud-based integration systems. I look forward to continuing to grow in customer empathy, product mentality, technical leadership, and engineering practice.\n","permalink":"https://blog.danieladamstech.com/2023/domain-allocation-trends/","summary":"I was thinking about how to best chronicle my career so far showcasing the variety of contributions I have delivered. After pondering, I came up with the idea of a graph showing my focus between the areas of software, data, cloud, and security.\nI had the look I wanted in mind but wasn\u0026rsquo;t sure how to implement it. My first concept was a stacked bar graph with variable width bars depending on my time in the position.","title":"Domain Allocation Trends"},{"content":"I have always been a nerd. As a kid, my first three big purchases with my own money were a pair of binoculars, Motorola walkie-talkies, and rechargeable batteries to power them. My brother and I played with those for years - at the house, at the park, and in many other settings. My introduction to ham radio was by reading Hardy Boys novels, including The Short-Wave Mystery. I also met someone who worked at Trans World Radio and showed me his mobile radio setup that could talk with South America.\nIn 2020, about 5 years after graduating college, I was looking for a hobby. The prior years had been busy with young kids and working hard on AWS and Red Hat certifications for work. Some friends were discussing what were the best radios for camping, and that triggered the thought I should get licensed and start playing with ham radio. After that, I channeled my energy into learning about radio. I couldn\u0026rsquo;t just be an amateur radio operator, I had to get the highest license class. I couldn\u0026rsquo;t just do any communication mode, I had to pick the most challenging one - CW or Morse code. I couldn\u0026rsquo;t just be an average CW operator, I had to get to the top of the speed distribution! :-)\nTimeline of Progress March 2020, literally a week before everything shut down due to COVID: I received my amateur radio license after passing the first two license levels (Technician and General) End of summer 2020: upgraded to Amateur Extra, the highest license class, when remote testing opportunities opened up February 2021: started learning Morse code on my own on the computer Fall 2021: joined Long Island CW Club and started learning to head copy without writing anything down January 2022: CW Academy Intermediate class with John N8AA, 20 words per minute at the end Summer 2022: CW Academy Advanced class with Phil NE0S, 30 words per minute at the end Stable speed throughout 2023 as I ramped down intense practice. I can converse at ~35-40 wpm and contest at ~30 wpm. Resources for Learning Morse Code Phase 1 - learning the letters (7 months) I started my Morse code journey using the website lcwo.net, which stands for \u0026ldquo;Learn CW Online.\u0026rdquo; This site will teach you how to recognize the letters and numbers and type them out as you hear them come in. It uses two strategies for making the learning process \u0026ldquo;stick\u0026rdquo; as much as possible. The \u0026ldquo;Koch Method\u0026rdquo; progresses a specific sequence of letters hitting distinctive sounds first. You start with two letters, learn those to a desired accuracy level (80-90%), and then add the next. This repeats until you know the full alphabet. The second technique, Farnsworth spacing, keeps the speed of the individual letters high and leaves more space between letters to give you a chance to think and get the letter written down. The worst thing you can do in learning Morse code is to count the \u0026ldquo;dits\u0026rdquo; and \u0026ldquo;dahs.\u0026rdquo; That will plateau your speed and block you from progressing.\nOne of my favorite parts about LCWO was the progress statistics it gathers. Seeing the visual indicator of improvement was motivating. You can see how in attempts #120-160, I ramped up my speed and my accuracy dropped. But then when I reset to a new baseline, my accuracy was higher than before. This parallels keeping a log for working out - something I have failed to do haha!\nMy first Morse code contact was at Field Day in June of 2021. It took me four hours to make a single contact. I had not anticipated how quickly people would be going in the contest. I thought the short exchange would make up for a bit more speed. Oops! However, I was determined to make the contact once I had committed to the idea. I tried for about an hour at the group site with the other club members. Then when I went home, the next day I tried for about another 3 to finally get it. I used a memory keyer with my call sign and exchange (what you send to the other station in a contest) programmed to make sure I wouldn\u0026rsquo;t mess up sending. This just goes to show it is never easy on your first attempt, but you can get through it! It just gets easier with each one after that.\nUpon reflecting on this phase, there were some things I did well and others I would have changed. I think I did a good job knowing where my proficiency target was and taking steps to work towards that. To that end, I did my LCWO practice at a character speed of 30 wpm with spacing between for an effective speed of 10. This helped me not count the dots and dashes, but I did still find myself repeating the letter sound to myself in the space between letters in order to interpret. That turned out to be a habit I needed to break later, but I didn\u0026rsquo;t let it go too far so it wasn\u0026rsquo;t too bad. In order to transition to getting on the air (targeting 15 wpm), I would bump the character speed down 2 for every effective speed bump up of 1. The goal was to meet in the middle.\nMy regret from this period in my journey was not getting on the air and having live practice with other people. I am a perfectionist, so I think this influenced my desire to have everything \u0026ldquo;not perfect but almost\u0026rdquo; before getting on the air. In reality, people are super supportive and gracious with someone starting out in their Morse code journey. They want you to get on the air and talk with them since they know that is the best way to make progress. I just did receiving practice these months, but should have done some sending as well as getting on the air. Receiving or \u0026ldquo;copying\u0026rdquo; is the part that takes the most work, but you don\u0026rsquo;t want to neglect sending either. Hearing yourself send with paddles and a keyer can give a helpful reinforcement cycle.\nPhase 2 - online classes and head copy (10 months) My next phase was trying to get on the air more and working towards head copy. Resources during this time included Long Island CW Club, CW Ops Academy, and Morse Code Ninja . LICW and CWA have video classes you join with other students working on learning Morse code. Morse Code Ninja by Kurt Zoglmann, AD0WE. Every page on his site contains helpful information, but the advice page is spot-on. I won\u0026rsquo;t repeat it all here, but the crux is to try to practice at the proficiency level you want to achieve, not what you can do now. Going too slow will cause you to hit plateaus. Going faster will seemingly take a while for it to \u0026ldquo;click\u0026rdquo;, but you\u0026rsquo;ll get where you want to go in the end.\nHead copy is like learning to touch type without looking at the keyboard. Practice is a binary option - either you look at the keyboard or you don\u0026rsquo;t. In Morse code, either you write it down or you don\u0026rsquo;t. As the saying goes, you have to get comfortable with uncomfortable practice. Just as touch typing provides huge benefits in the workplace, head copying Morse code will make the whole experience much more relaxed and enjoyable than trying to scramble all the time worried you won\u0026rsquo;t be able to write fast enough.\nThe program that helped me the most in reaching my CW goal was CW Academy , hosted by the organization CWOps. Their program facilitates a strong level of immersion and causes real gains. I had the goal coming out of the advanced class to be at 25 words per minute (the benchmark for joining the club), but I passed that and was comfortable at 30! CW Academy runs 3 times a year for an 8 week semester with classes twice a week. There are daily assignments between classes. That intense sprint was a good model for me to level up in each class.\nHead copy allowed me more opportunities for practice. Instead of getting in 15 minutes a day at the keyboard, I figured out ways to fit in almost an hour a day using the Morse Ninja playlists. I would listen to those while doing my morning exercise or doing the dishes at night.\nPhase 3 - conversational speed practice (6 months) Coming out of the CW Academy classes, I was doing fine in all practical purposes. However, I wanted to be the best that I could be and wanted to continue to get better. One benefit you get at higher speeds is conversation gets more comfortable. The average person speaks at around 120 words per minute. You\u0026rsquo;re not going to match that with Morse code, but 35 wpm is decidedly more enjoyable to have a longer conversation than even 25. Some people like Steve WF3T will use computers generating Morse code from keyboard input to chat in the 40s and 50s wpm. Audio for these can be found at streammyaudio.com archive with titles CRSnet.mp3.\nThe way I worked on this speed was by reading ebooks in Morse code. I downloaded copyright-free books from Project Gutenberg and converted them to Morse code. I edited them to remove extraneous punctuation and slang ('an to and) and stored them in this Github repo for anyone else to use. I would convert these text files to mp3s with ebook2cw by Fabian Kurz, a German ham. Those mp3s were converted into an RSS feed using a script by Amine Sehili. I would start up a web server with the RSS XML and audio files on my laptop and download them to Apple podcasts on my phone for offline (when the laptop web server was turned off) listening. All my listening was done with cheap Bluetooth hard plastic earbuds. Forgoing any attempt at sound isolation allowed me to be more present in the real world while Morse code was playing and to train my brain to decode through background noise.\nIn all, I practiced approximately 6 months of 15-20 minutes a day, a year of an hour a day, and then 6 months of 30 minutes a day. Mid 2023, I started writing this blog and Taekwondo, so my Morse code practice went down to around maintenance mode and using it for fun. I\u0026rsquo;ll sometimes listen to high-speed roundtable conversations by WF3T and his friends. By far, the best way to improve and maintain skills is to be on the air using them. When learning, I tried to be on the air every day. Now, my goal is 2-3 times a week. The graph below shows when automated \u0026ldquo;spotting\u0026rdquo; systems heard me calling out for radio contacts.\nWishing you the very best on your own Morse code journey!\n","permalink":"https://blog.danieladamstech.com/2023/on-learning-morse-code/","summary":"I have always been a nerd. As a kid, my first three big purchases with my own money were a pair of binoculars, Motorola walkie-talkies, and rechargeable batteries to power them. My brother and I played with those for years - at the house, at the park, and in many other settings. My introduction to ham radio was by reading Hardy Boys novels, including The Short-Wave Mystery. I also met someone who worked at Trans World Radio and showed me his mobile radio setup that could talk with South America.","title":"On Learning Morse Code"},{"content":"This is the last post chronicling the reading of the book Influence by Dr. Robert Cialdini. The first was an introduction and question of influence vs. persuasion, and the next covered the first four \u0026ldquo;weapons of influence\u0026rdquo;: contrast, reciprocation, commitment plus consistency, and social proof. Continuing on, here we cover liking, authority, and scarcity.\nChapter 5: Liking It is better to be liked than not liked. Surprise! But what types of liking are there? The book lists five: physical attractiveness, similarity, compliments, contact plus cooperation, and association. The various \u0026ldquo;liking\u0026rdquo; factors can produce a halo effect where one aspect of a person is noticed and prevents the observer from seeing a more holistic picture. The halo effect is one of the performance review biases covered in Gergely Orosz\u0026rsquo;s The Software Engineer\u0026rsquo;s Guidebook. I just received my copy last week! I need to finish up a couple of other in-progress books on my book list, and then I will read that one.\nWorking towards positive outcomes with teamwork leads to liking. As you can imagine, two cabins of boys at camp can develop a rivalry, especially if there are competitive games between the groups. Once this competitive spirit has grown, it can lead to negative connotations and attacks on the personal character of the other group. A research study let this feeling mount and then tested how effectively it could be reversed. To trigger the reversal, the researchers constructed scenarios where the two groups of boys needed to work together. These situations included the food transport truck being stuck and an issue with the water pipes. Both situations lead to the groups working together to come to a solution.\nIn the corporate world, related but separate teams can come together to accomplish something and build interpersonal bonds. Many times, different teams working on the same project are approaching it from different angles. This leads them to value different things on the way towards the common goal. Team-building activities can proactively strengthen relationships and lead to higher levels of empathy for counterpart team\u0026rsquo;s objectives and methods. When I was on the Cloud/Data Engineering team, the Data Engineering team and the Enterprise Data Management Office team volunteered together several times at the local food bank. Now I\u0026rsquo;m on the Strategic Customer Technology (Integrations) team. Just last week we did a full-day Habitat for Humanity house build with our business-side counterpart, the Customer Solutions team.\nChapter 6: Authority The Milgram study showed surprising results of compliance with authority, even when obeying caused physical pain to another person. The subject of the experiment played the role of a \u0026ldquo;teacher\u0026rdquo; who would deliver increasingly strong shocks to the \u0026ldquo;learner\u0026rdquo; when an incorrect response was given. The most enlightening data point was the inaccuracy of estimation of at what point the \u0026ldquo;teacher\u0026rdquo; would stop following the researcher\u0026rsquo;s instructions. A group of \u0026ldquo;colleagues, graduate students, and psychology majors at Yale University (where the experiment was performed)\u0026rdquo; estimated that only 1-2 percent of people would continue on to the end. As it turned out, no one stopped before 2/3 of the way through the experiment. 65% continued all the way to the end, even when the \u0026ldquo;learner\u0026rdquo; stated he had a heart condition.\nMethods for claiming authority include titles, clothes (security guard uniform, business suit), and other trappings like exotic cars. The book treated each of these individually, but one thing I wonder is how closely they are linked as a proxy for wealth and social status. What is the difference in the behavior of a subject in response to an authority figure with direct influence on the subject vs with respect to a \u0026ldquo;powerful\u0026rdquo; high social class figure where there is no direct line of authority?\nJames 2:2-4 calls out this dishonorable tendency to defer to wealthy people based on their clothing:\nFor if a man wearing a gold ring and fine clothing comes into your assembly, and a poor man in shabby clothing also comes in, and if you pay attention to the one who wears the fine clothing and say, “You sit here in a good place,” while you say to the poor man, “You stand over there,” or, “Sit down at my feet,” have you not then made distinctions among yourselves and become judges with evil thoughts?\nChapter 7: Scarcity All these principles of influence trace back to shortcuts our mind uses to make decision-making easier. Unless manipulated, they are usually a proxy of logical decision processes. We all are familiar with the relationship between supply and demand. Thus we associate scarcity (low supply) as driven by high demand. On a slightly different angle taking into account both supply and demand, if many others are buying the item, we assume it is priced correctly on the supply/demand curve (or is the efficient frontier in investing.)\nThis manipulation can cause someone to ignore the remainder of the value proposition and focus too much on the proxy of availability. During college, the author\u0026rsquo;s brother made money by selling cars. But the fascinating part was he bought and flipped them through the same newspaper\u0026rsquo;s classified section. How could he consistently sell the car for more than he had paid for it? The secret was scheduling all prospective buyers to come at the same time. This caused a heightened sense of scarcity which would not have been present with individual showings. When two or three people showed up, he would line them up by order of arrival and say they must wait until the person ahead of them in line had made his or her purchase decision. This added \u0026ldquo;now or never\u0026rdquo; pressure to the situation and would cause even partially interested buyers to pay the full asking price for fear the person behind them in line would take the deal if they walked away.\nOne signal indicative of whether one should completely disregard scarcity is the contrast between owning something rare vs using it. The latter does not change whether an item is rare or plentiful. We should step back and ask ourselves the purpose of buying a particular item, and if we should be influenced by scarcity. I\u0026rsquo;m an amateur radio operator, and some hams can border on collectors. For one person, having a first-ten serial number or a limited edition Morse code paddle means something. Another person may only be concerned about the performance of the item. Knowing that you fall into the second camp can help you attempt to distance yourself from scarcity influence.\nThis book has been an intriguing read. It put in words some subconscious patterns I had seen before and enlightened me to others I had not. Now I am more able to spot these influence tactics and also decide when to ethically employ them myself. These strategies do not have to be applied for purposes that are not in the best interest of the subject. If I built a new platform component I want other development teams to adopt, highlighting each time a team adopts it exhibits social proof that can snowball into more adoption. Be influential; be responsible!\n","permalink":"https://blog.danieladamstech.com/2023/influence-book-part-3/","summary":"This is the last post chronicling the reading of the book Influence by Dr. Robert Cialdini. The first was an introduction and question of influence vs. persuasion, and the next covered the first four \u0026ldquo;weapons of influence\u0026rdquo;: contrast, reciprocation, commitment plus consistency, and social proof. Continuing on, here we cover liking, authority, and scarcity.\nChapter 5: Liking It is better to be liked than not liked. Surprise! But what types of liking are there?","title":"Influence Book Part 3"},{"content":"Previously I shared a post Staff Engineer Resources that included a collection of books, podcasts, and blogs I found helpful. That list was a snapshot in time, but I wanted to create a living page to track my reading. Both improving my writing and increasing my book reading are personal improvement goals of mine.\nI like the \u0026ldquo;Reading,\u0026rdquo; \u0026ldquo;Read,\u0026rdquo; and \u0026ldquo;To Read\u0026rdquo; sections in Goodreads, but wanted a book list that fit the look and feel of the site more than their embedded widget. This made me think of the Gatsby static site generator , which was built from the ground up to pull data in from other sources. A quick Google showed Goodreads actually deprecated their API in 2020, several years ahead of Reddit and Twitter making similar moves recently in response to LLM training. My new book list/reading tracker page is in the navigation bar links above!\n","permalink":"https://blog.danieladamstech.com/2023/book-list-project/","summary":"Previously I shared a post Staff Engineer Resources that included a collection of books, podcasts, and blogs I found helpful. That list was a snapshot in time, but I wanted to create a living page to track my reading. Both improving my writing and increasing my book reading are personal improvement goals of mine.\nI like the \u0026ldquo;Reading,\u0026rdquo; \u0026ldquo;Read,\u0026rdquo; and \u0026ldquo;To Read\u0026rdquo; sections in Goodreads, but wanted a book list that fit the look and feel of the site more than their embedded widget.","title":"Book List Project"},{"content":"Reading The Staff Engineer\u0026rsquo;s Path: A Guide for Individual Contributors Navigating Growth and Change Never Split the Difference: Negotiating as if Your Life Depended on It Read Engineering Designing Data Intensive Applications (pre-blog era) The Software Engineer\u0026rsquo;s Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups - blog post Engineering Leadership Staff Engineer: Leadership beyond the management track - blog post Team Topologies - blog post Influence: The Psychology of Persuasion - blog post 1, part 2, part 3 To Read How to Win Friends and Influence People The Engineering Executive\u0026rsquo;s Primer: Impactful Technical Leadership Kill It with Fire: Manage Aging Computer Systems These books can also be found on my Goodreads page. ","permalink":"https://blog.danieladamstech.com/book-list/","summary":"Reading The Staff Engineer\u0026rsquo;s Path: A Guide for Individual Contributors Navigating Growth and Change Never Split the Difference: Negotiating as if Your Life Depended on It Read Engineering Designing Data Intensive Applications (pre-blog era) The Software Engineer\u0026rsquo;s Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups - blog post Engineering Leadership Staff Engineer: Leadership beyond the management track - blog post Team Topologies - blog post Influence: The Psychology of Persuasion - blog post 1, part 2, part 3 To Read How to Win Friends and Influence People The Engineering Executive\u0026rsquo;s Primer: Impactful Technical Leadership Kill It with Fire: Manage Aging Computer Systems These books can also be found on my Goodreads page.","title":"Book List"},{"content":"In a microservices environment where services align to team boundaries, many times it will be necessary to call a service owned by another team. My team is responsible for the customer-facing external API layer for B2B transactions. We maintain multiple versions of MISMO APIs and handle security, logging, monitoring, and mapping the customer request to a standardized data format for our internal APIs. The APIs we are fronting do a lot of work. They run large rule sets and bookkeeping for mortgage insurance rate quotes or orders. Some are even dependent on external APIs like a credit pull. This leads to variance in response time.\nAs the customer-facing team, we want to ensure all our customers have a good experience. This leads us to track the overall system response time. However, it is hard to track the performance of our own code via that metric as our code accounts for less than 5% of the total transaction time. How then can we set up meaningful alerts for our team\u0026rsquo;s performance? We need to track timing at the team microservice boundary. This is simple when you\u0026rsquo;re the \u0026ldquo;leaf node\u0026rdquo; on the downstream side. However, the entrypoint layer takes time on the front end and back end of the transaction. We need to identify a span in the distributed trace that represents the \u0026ldquo;noise\u0026rdquo; in measurement that we want to remove and then subtract that duration from the outer span. This requires a bit of playing around with platform-specific query capabilities. In batch SQL, options are not very constrained, but sometimes approach needs to be adapted a bit to fit a streaming data analysis pipeline like the one that drives monitoring alerts.\nThe query below attempts to execute this strategy. The where clause pulls in both the outer and inner transactions into the intermediate result set. Then the filter() portion calculates two metrics from the bisection of that result set by microservice and figures the difference.\n1 2 3 4 5 6 FROM Span SELECT filter(percentile(duration, 90), where appName = \u0026#39;Entrypoint Layer\u0026#39;) - filter(percentile(duration, 90), where appName = \u0026#39;Last Hop to Include\u0026#39;) as entrypointLayerDuration WHERE (appName = \u0026#39;Entrypoint Layer\u0026#39; and TransactionCustomAttributeName = \u0026#39;CustomValue\u0026#39;) or (appName = \u0026#39;Last Hop to Include\u0026#39; and name = \u0026#39;External/destination.corp.net/CommonsHttp/execute\u0026#39;) This query works as long as the first and last hop queries can uniquely retrieve transactions under measurement. For example, if two different transactions in the last hop microservice made a call to the same uninstrumented DNS name, the second half of the query would pull in more traces and skew the timing. Another situation where this query would have a problem is when the entrypoint service errors and quickly returns. This could be due to invalid data format, bad auth, etc. That drives down the outer duration to nearly instantaneous while the inner duration only accounts for transactions that made it all the way to the last hop service and remains as usual. The metric could go negative in that case.\nNewRelic recently rolled out subquery joins. which can help combat that issue. I tested this query with a mix of all-the-way-through and short-circuited transactions, and it does give accurate results. While the other query can cover a long timeseries, this one is relatively limited due to what I\u0026rsquo;m assuming is the cardinality requirement of the join. It is not allowed for an alert query, even without limit max. The explicit limit max is required in the ad-hoc query to get the full result set or only about 10 rows will be returned. I\u0026rsquo;ll continue to work with NewRelic to figure a workaround on how to get this working in alerts. I might end up falling back to creating another custom attribute on the entrypoint service that allows me to only select the desired traces for the filter() style query.\n1 2 3 4 5 From Span JOIN (FROM Span SELECT trace.id , duration as innerDuration where appName = \u0026#39;Last Hop to Include\u0026#39; and name = \u0026#39;External/destination.corp.net/CommonsHttp/execute\u0026#39; LIMIT MAX) on trace.id SELECT percentile(duration - innerDuration, 90) as entrypointLayerDuration WHERE (appName = \u0026#39;Entrypoint Layer\u0026#39; and TransactionCustomAttributeName = \u0026#39;CustomValue\u0026#39;) LIMIT MAX ","permalink":"https://blog.danieladamstech.com/2023/measuring-entrypoint-layer-latency-in-newrelic/","summary":"In a microservices environment where services align to team boundaries, many times it will be necessary to call a service owned by another team. My team is responsible for the customer-facing external API layer for B2B transactions. We maintain multiple versions of MISMO APIs and handle security, logging, monitoring, and mapping the customer request to a standardized data format for our internal APIs. The APIs we are fronting do a lot of work.","title":"Measuring Entrypoint Layer Latency in NewRelic"},{"content":" Influence by Dr. Robert Cialdini covers some useful and intriguing psychological constructs that shape how we act in society. As I mentioned in my previous post, one must take care in applying ethics to the situation and analyzing which subset of tactics is acceptable.\nBelow I walk through the book by chapter highlighting topics and examples that I found interesting.\nChapter 1: Weapons of Influence One theme this book exposes is the subconscious things we would be surprised influence us but actually do. There are empirical studies throughout the book supporting the various points. Take these three requests in the context of waiting in line for a copier and asking to skip to the font.\n\u0026ldquo;Excuse me, I have five pages. May I use the Xerox machine because I\u0026rsquo;m in a rush?\u0026rdquo; 94% success \u0026ldquo;Excuse me, I have five pages. May I use the Xerox machine?\u0026rdquo; 60% success \u0026ldquo;Excuse me, I have five pages. May I use the Xerox machine because I have to make some copies?\u0026rdquo; 93% success One would assume that being in a rush would lead to a greater chance of skipping in line, but it really was the trigger word because.\nContrast principle: much more likely to purchase a $95 sweater after purchasing a $495 suit than the other way around. The same thing applies to realtors showing overpriced, dilapidated properties leading up to the \u0026ldquo;good\u0026rdquo; deal they want to sell their clients on. The contrast fresh in the client\u0026rsquo;s mind makes him or her see the value of the final house after the setup sequence.\nAnother theme of the book is that humans use pattern matching and take shortcuts to make sense of the world. For example, expensive=good. If we analyzed everything from first principles every time, we would function at the speed required to make decisions in normal life. The obvious observation this raises is we generally think of stereotyping as bad, but this emotionless observation says it is required to function in life. I have some questions. Is there something else we can categorize certain learning patterns as? Pattern-matching? Inference? Or is it all just plain old stereotyping? Inference conjures the thought of machine learning models and generative AI. Generating Barbies from various countries exposed the stereotypes and racism unchecked inference can lead to.\nChapter 2: Reciprocation Reciprocation is a phenomenon that is common in American culture. When someone helps us out, we want to return the favor and help them in return. This can be a solid foundation of good working relationships and even friendships, or it can be manipulated out of balance. This is because the initiator chooses the both first favor and the requested return favor. The indebtedness can be exploited when the return favor is much larger than the original.\nIn Luke 14:12-14, Jesus says:\nHe said also to the man who had invited him, “When you give a dinner or a banquet, do not invite your friends or your brothers or your relatives or rich neighbors, lest they also invite you in return and you be repaid.\nBut when you give a feast, invite the poor, the crippled, the lame, the blind, and you will be blessed, because they cannot repay you. For you will be repaid at the resurrection of the just.”\nA variation on the reciprocation of favors is reciprocal concession. In this scenario, the initiator does not give anything to start the process. He simply makes a large request, knowing the answer will probably be \u0026ldquo;no.\u0026rdquo; The reciprocation comes into play with the follow-up request. The initiator\u0026rsquo;s concession down to the smaller request will make the subject feel pressured to return the \u0026ldquo;favor\u0026rdquo; and concede by agreeing to the second proposition.\nMy kids have figured out this principle already! They will sometimes ask to bring a toy into whatever destination we are driving to. When I say no, they then ask to bring it but leave it in the van. I\u0026rsquo;m more likely to say yes due to the reciprocal concession principle. If they had led with their second request, I would have more likely said a flat no.\nHere is the defense angle: don\u0026rsquo;t eschew all favors as some will be genuine. Pay attention to if the requested reciprocation is substantially larger than the first act or if negotiation ensues \u0026ldquo;walking down\u0026rdquo; to smaller and smaller concessions. Then you know it was a ploy from the beginning and do not need to feel indebted to repay what was given with ulterior motives.\nChapter 3: Commitment and Consistency Continuing down a sub-optimal path due to resources already invested is the definition of sunk cost fallacy. This can be a risk in software engineering. One quote says that good engineers are the ones willing to throw away their code when the opportunity arises. Sunk cost is a commitment to previous actions, but the book broadens the concept to also include consistency with previous statements.\nOne case study was how to keep toy sales up after big sale seasons like Black Friday and Christmas. Toy companies sometimes use this sequence to make post-Christmas sales.\nLaunch strong ad campaigns for a specific toy during the Christmas shopping season Parents will make commitments to kids to get that specific toy Undersupply that toy to stores so it is not available during holiday shopping Run more ads in January to remind kids of the previous commitments Parents, wanting to stay true to their previous commitments, will buy said toy even though they already bought something else for the kids\u0026rsquo; presents. Edgar Schein studied Some Observations on Chinese Methods of Handling Prisoners of War (JSTOR) and also cited in the Defense Technical Information Center. Some of his findings were incorporated in this chapter.\nThe Chinese communist POW camps tried to indoctrinate prisoners via the \u0026ldquo;lenient policy.\u0026rdquo; The program focused on getting soldiers to make concessions about how things were not perfect back home, and maybe Communism wasn\u0026rsquo;t totally wrong. The prisoners were then coerced to write these statements down. Written evidence helped solidify it in their own minds as well as spread the statements to others who were more resistive (see chapter 4 - social proof.) Another avenue for collecting these statements was political essay contests. They had various ways of encouraging participation - small prizes, middle-of-the-road essays winning just enough to keep people hopeful but still force some concessions.\nTraits that make commitments the most effective are being active, public, and effortful (page 92.) A detracting factor is too much external motivation. Minimizing external pressure like large prizes or shaming makes someone own their decision in a stronger way. This is because they cannot blame any other reason for their commitment other than it\u0026rsquo;s what they wanted to do and \u0026ldquo;that\u0026rsquo;s who they are.\u0026rdquo; This strategy is exemplified in both the POW camp essay choice of prizes as well as modern-day fraternity initiations. The essay contest prizes were small, only a couple of cigarettes or pieces of fruit. Fraternities keep initiation hazing separate from their community service so there is no \u0026ldquo;mental out\u0026rdquo; that an applicant can say he did it for a cause other than he wanted to join. On the pressure and shaming side, it has been shown that kids will more deeply internalize parental guidelines if strong threats of punishment are not given. Wariness of punishment only applies if they think they might get caught. Knowing something is inherently the right thing to do is trickier to convince of, but results in stronger adherence.\nChapter 4: Social Proof Social proof is another shortcut our brain takes when trying to determine the correct action. Learning is picking up patterns in the world and internalizing them. Sometimes that is from first principles, and others it is by observation. The majority of the time, this is a valid learning and reaction mechanism. However, manipulated responses or cascading blind following can make the heuristic incorrect.\nHarmless examples of social proof include laugh tracks in comedy TV, claqing in opera houses, and salting tip jars with dollar bills instead of coins. Even when people are aware of those practices, they are still likely to be influenced by them. Unfortunately, very negative examples also exist. Suicide rates peak after publicized suicide stories on the news. Jonestown was a horrific example of how a leader did not have to convince everyone individually, only enough to start a tidal wave of compliance.\nA practical phenomenon we will probably run into in our daily lives is pluralistic ignorance. This is where signs exist that something is not right, but bystanders take cues from other bystanders to go on their way and not act. They suppose from the inaction of others that they are misinterpreting the signs of trouble and everything is actually fine. This makes me think of circular dependencies in software. The more public the situation, the less likely the victim is to receive help. When there is only one witness, it is clear that a specific person needs to act. A larger number of witnesses leads to more ambiguity as to who should act. The way to counter this tendency is to assign a specific task to a specific person. The example from the book is, \u0026ldquo;You, sir, in the blue jacket, I need help. Call an ambulance.\u0026rdquo;\nDefenses against illegitimate use of social proof take two forms. First, try to spot falsified actions of social proof. This includes paid audience members, paid testimonials in an advertisement (even if that person looks like you), and modern-day \u0026ldquo;influencers.\u0026rdquo; Sometimes the actions taken by the crowd will not be fraudulent, just misguided. This leads to the second defense, spotting snowballing errors. An example of this is when multiple drivers change lanes at the same time on a highway, especially during rush hour. If visibility is bad up ahead, cars behind them can try to follow suit thinking there is an obstruction in the road up ahead. Ironically, cars merging over to avoid an anticipated wreck up ahead can cause an actual wreck.\nA fascinating situation where \u0026ldquo;following the crowd\u0026rdquo; is the logical thing to do is during a bank run. SVB Bank fell to this wave of withdrawals as panic was spreading on Twitter. Bank runs are a tragedy of the commons situation. The only way to quell a bank run is for everyone to stop drawing money and let the bank stabilize. But the undeniable self-preserving move is to withdraw one\u0026rsquo;s own money while it is still accessible. Most times in a snowballing behavior situation the best thing to do is not to get sucked in, but this is one exception. What are others?\nThis post covers the first half of Influence by Dr. Robert Cialdini. I will cover the remainder in another post coming up! Thanks for reading, and I hope you found some of the tidbits interesting.\n","permalink":"https://blog.danieladamstech.com/2023/influence-book-part-2/","summary":"Influence by Dr. Robert Cialdini covers some useful and intriguing psychological constructs that shape how we act in society. As I mentioned in my previous post, one must take care in applying ethics to the situation and analyzing which subset of tactics is acceptable.\nBelow I walk through the book by chapter highlighting topics and examples that I found interesting.\nChapter 1: Weapons of Influence One theme this book exposes is the subconscious things we would be surprised influence us but actually do.","title":"Influence Book: Part 2"},{"content":"I listened to an older Software Engineering Podcast episode with Josh Wills this morning while running. Hearing from the first director of data engineering at Slack was insightful and engaging with some of his spicy takes! A big idea is that approaches that look opposite on the surface can both work out if executed well. We should learn from others\u0026rsquo; experiences and not disregard them since they don\u0026rsquo;t line up with ones that have brought us success personally.\nOne topic was the path of decisions built on foundational mental models and paradigms. Google and Facebook are both very smart and successful companies that approached the issue of logging from completely different angles. When engineers were within their respective companies, the approach was standard and almost taken as a priori due to the consistency behind it. But then when both groups converged at Slack, the mostly-Google approach won. It won not because it was better, but because the first hires came from that approach and had first-mover status. Nothing would have been gained by refactoring between two \u0026ldquo;equally good\u0026rdquo; but very different approaches.\nIn the \u0026ldquo;choose your own adventure\u0026rdquo; game of the next 10 years of use within Google and Facebook, the two different approaches would run into different issues. However, both sets of issues could be overcome by good engineering and addressing pain points as they came up. Two sides of the coin are that the decision had a huge impact on the day-to-day lives of engineers for years to come (in different ways), but either decision would be overcome-able even if it wasn\u0026rsquo;t 100% optimal. A point brought up along those lines is for CTOs at early-stage companies laying the groundwork, the 95% of decisions they got right (enabling the company to still be a company) aren\u0026rsquo;t felt as much as the small sliver that causes pain. Analogy: no one was thinking about airplane cockpit door locks on 9/10.\nGoogle Logging\nSingle Protobuf schema for all logs across all services (huge!) One log at the end of a transaction Highly denormalized data, processed with MapReduce Aligned to the business of search (not really transactional, more about data aggregation) Facebook Logging\nNo single massive schema, multiple smaller Thrift schemas Multiple logs per transaction along the way Smaller logs joined by relational query engines - Hive came from Facebook Aligned to stack (PHP/MySQL) and business (more of transactional with posts) ","permalink":"https://blog.danieladamstech.com/2023/multiple-successful-paths/","summary":"I listened to an older Software Engineering Podcast episode with Josh Wills this morning while running. Hearing from the first director of data engineering at Slack was insightful and engaging with some of his spicy takes! A big idea is that approaches that look opposite on the surface can both work out if executed well. We should learn from others\u0026rsquo; experiences and not disregard them since they don\u0026rsquo;t line up with ones that have brought us success personally.","title":"Multiple Successful Paths"},{"content":"In software development, we generally must use dependencies to efficiently get more complex functionality we don\u0026rsquo;t want to write ourselves. However, there are some types of dependencies that prove to be more painful when maintaining software than others. Fewer dependencies leads to less upgrade work including less security vulnerability remediation work. Below I rank categories of dependencies with respect to the likelihood to have coupling issues during framework upgrades.\nThe Dependencies Good: broadly adopted, well-maintained dependencies that provide necessary functionality orthogonal to the framework (Spring Boot) AWS SDK , MapStruct , JAXB used in another project Not ideal, but necessary: coupled to the framework, but necessary functionality SpringFox for OpenAPI spec generation Probably not necessary: small uses of convenience functions or premature optimizations Pulling in the full Apache Commons Collections just for CollectionUtils.isNotEmpty(), explicitly overriding to use Apache HTTP client instead of the Spring Boot default for a low-traffic service Bad: tightly coupled to the framework but not officially supported by the framework team Derjust Spring Data DynamoDB stopped being maintained but required major code reorganization to remove. The Story I was upgrading a small service that persisted configuration data in DynamoDB and called a couple of upstream APIs. It was developed and deployed to prod, but never fully adopted. A couple of years later, there were new requirements and changes needed to make it usable. Upgrading this service gave me an ah-ha moment regarding types of dependencies and the level of upgrade risk they bring with them. After glancing at the original state of the service, here were my goals: Upgrade Spring Boot 2.0.6 to 3.1.x, Java 8 to 17, remove unofficial spring-data-dynamodb. , and upgrade AWS Java SDK v1 to v2. There wasn\u0026rsquo;t any issue with AWS SDK v1, but I wanted some experience with v2 since that\u0026rsquo;s what we\u0026rsquo;ll be using for net-new services.\nOnly after I had started bumping Spring Boot versions did I realize that the unofficial Spring Boot DynamoDB dependency was last released in January of 2019 (four and a half years ago.) That release only supported up to Spring Boot 2.1. Spring Boot 2.5 was the first release to support Java 17. The upgrade path would have been to remove Spring Data DynamoDB, upgrade to Spring Boot 2.5, Java 17, Spring Boot 3.1. Along the way would have been a Junit 4 to 5 migration and SpringFox to SpringDoc. The code had a good bit of extra dependencies, boilerplate copy/paste servlet filters and loggers, etc. All those together led me to write a minimal service from scratch on Spring Boot 3.1 and Java 17.\nI utilized MapStruct to map between the API data format of shared properties + overrides and the denormalized set of records in DynamoDB. I wanted to do as much as possible with immutable Java records instead of Lombok-annotated data classes. The DynamoDB enhanced client came through and was compatible with Records following the same pattern this documentation shows with classes.\nThe Code DynamoDB mapper immutable entity\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 @DynamoDbImmutable(builder = DemoEntity.DemoEntityBuilder.class) @Builder(toBuilder = true) public record DemoEntity( // keys and special DynamoDB attributes @DynamoDbPartitionKey String pk, @DynamoDbSortKey String sk, @DynamoDbVersionAttribute Long version, // record level lock // record attributes String shared_attribute_one, String shared_attribute_two, String shared_attribute_three, String override_attribute_one, String override_attribute_two ) { } DTO to entity mapper\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 @Mapper(componentModel = \u0026#34;spring\u0026#34;) public interface RootMapper { /** * Singleton mapper, combine root level parameters with leaf level parameters. */ // keys @Mapping(source = \u0026#34;root.id\u0026#34;, target = \u0026#34;pk\u0026#34;) @Mapping(expression = \u0026#34;java(\\\u0026#34;SORT_KEY_PREFIX#\\\u0026#34; + leaf.leafType())\u0026#34;, target = \u0026#34;sk\u0026#34;) // root level attributes @Mapping(source = \u0026#34;root.sharedAttributeOne\u0026#34;, target = \u0026#34;shared_attribute_one\u0026#34;) @Mapping(source = \u0026#34;root.sharedAttributeTwo\u0026#34;, target = \u0026#34;shared_attribute_two\u0026#34;) @Mapping(source = \u0026#34;root.sharedAttributeThree\u0026#34;, target = \u0026#34;shared_attribute_three\u0026#34;) // leaf level attributes @Mapping(source = \u0026#34;leaf.overrideAttributeOne\u0026#34;, target = \u0026#34;override_attribute_one\u0026#34;) @Mapping(source = \u0026#34;leaf.overrideAttributeTwo\u0026#34;, target = \u0026#34;override_attribute_two\u0026#34;) DemoEntity dtoToEntitySingle(Root root, Leaf leaf); /** * Combine the root level parameters with leaf level parameters. * Pass combo to MapStruct mapper that just has to worry about the singleton case. */ default List\u0026lt;DemoEntity\u0026gt; dtoToEntityList(Root root) { return root.leaves().stream() .map(leaf -\u0026gt; dtoToEntitySingle(root, leaf)) .toList(); } } ","permalink":"https://blog.danieladamstech.com/2023/framework-ish-dependencies/","summary":"In software development, we generally must use dependencies to efficiently get more complex functionality we don\u0026rsquo;t want to write ourselves. However, there are some types of dependencies that prove to be more painful when maintaining software than others. Fewer dependencies leads to less upgrade work including less security vulnerability remediation work. Below I rank categories of dependencies with respect to the likelihood to have coupling issues during framework upgrades.\nThe Dependencies Good: broadly adopted, well-maintained dependencies that provide necessary functionality orthogonal to the framework (Spring Boot) AWS SDK , MapStruct , JAXB used in another project Not ideal, but necessary: coupled to the framework, but necessary functionality SpringFox for OpenAPI spec generation Probably not necessary: small uses of convenience functions or premature optimizations Pulling in the full Apache Commons Collections just for CollectionUtils.","title":"Framework-ish Dependencies"},{"content":"I was given the book Influence by a coworker to help me work on leadership skills. She warned me that the book bordered on the social engineering side of influence. Boy, was that correct! The book goes into our natural instincts and reactions to others\u0026rsquo; behavior and how those patterns can be leveraged. One thing I thought was interesting and underscored the real-life applications of these theories were the studies referenced in the book. Experiments and quantitative data collection proved these theories out empirically.\nI had anticipated using the ideas from this book in an \u0026ldquo;offensive\u0026rdquo; capacity, but after reading the first couple of chapters, I feel they might be better suited for me personally on the \u0026ldquo;defensive\u0026rdquo; front. I am definitely a pragmatist in many respects. My courses of study and hobbies all lean towards the practical side: computer science in college, certifications after graduation, running/hiking/biking, ham radio, and recently Taekwondo. In college, my honors friends would joke with me about being a Philistine since many of them were English and history majors and appreciated historical philosophy in a way I could not.\nOne area of life where this is not true is in my interpersonal interaction. I lean pretty far towards the idealist side in personal relationships. I don\u0026rsquo;t know if this correlates to my being an optimist, but I want to assume the best about people. Even though the myth of meritocracy states that meritocracy isn\u0026rsquo;t broadly applicable in societal movements, my goal is to keep work decision-making discussions as close to the actual issues and tradeoffs as I can. Deep down, I want to avoid veering off into what I would consider side-channel attacks of the more manipulative psychological influence tactics. In many ways I am \u0026ldquo;an old soul\u0026rdquo;, but I think this might be an area where I have a sliver of an idealistic young person.\nKnowledge and judicious use of these strategies is a good thing. For example, one natural human pattern is reciprocation. I asked for some help with a Maven dependency resolution issue on Wednesday and was happy to pair with that same person when on Friday he had a Python Oracle database driver issue in a Lambda function. That reciprocation is just good teamwork. Where I might be aware of techniques and choose not to use them is giving out unrequested and unwanted small favors to foster a sense of indebtedness that will later be cashed in as a large return favor. People will naturally want to reciprocate with the same or greater proportions in order to clear the account with the other person. This fact can be used to selfishly extract favors from others. Awareness of the repertoire of techniques allows you to play better defense and not be taken in by them.\nSo, the big question\u0026hellip; As I am trying to ethically improve my influence as a whole, is there another word that captures the essence of my focus area? Persuasion?\nInfluence over Persuasion: Influence is more broad. Solely persuasion without motivation does not prompt action; no goals are actually accomplished. Persuasion over Influence: Persuasion (at least my connotations) leans into the actual issues being discussed in a setting where both parties are evaluating value judgments regarding a decision and coming to a consensus. It is not as affected by external factors like power differential. In my next post, I will dive into the contents of the first half of this book. These include contrast, reciprocation, and commitment to consistency.\n","permalink":"https://blog.danieladamstech.com/2023/influence-or-persuasion/","summary":"I was given the book Influence by a coworker to help me work on leadership skills. She warned me that the book bordered on the social engineering side of influence. Boy, was that correct! The book goes into our natural instincts and reactions to others\u0026rsquo; behavior and how those patterns can be leveraged. One thing I thought was interesting and underscored the real-life applications of these theories were the studies referenced in the book.","title":"Influence or Persuasion?"},{"content":"\u0026hellip;Or how good ideas sometimes come in the middle of the night.\nAn episode of the Big Ideas in App Architecture podcast introduced me to a set of talks by Rich Hickey. I think it was this one with Joe Lynch? Rich\u0026rsquo;s talks were Simple Made Easy and Hammock Driven Development . Both were insightful regarding what mindset to invoke when creating software.\nThe premise of hammock driven development is that thinking about a topic over a long enough period of linear time causes the subconscious mind to prioritize it. Prefer a week or more in the conscious thought space; it won\u0026rsquo;t happen with cramming for a day or two. That conscious prioritization lets your mind know it is something you want it to work on in the background. That can lead to an aha moment when you\u0026rsquo;re not expecting it. An implication of the timeline is that you should start thinking about projects and problems on the horizon with a good bit of lead time before you need to implement a solution. This will give you enough time to really digest the options. Who knows, this could all be a bunch of hooey. I tend to do my best thinking in the morning before the kids get up and before coming into the office since the house is quiet. I was one of those college students who would use the quiet study zone in the library to do my math homework. Today, I woke up at 3:45 with a generic idea of how the API I was trying to refactor could interact in a generic way with the data mapper to denormalize into the desired DynamoDB object structure. I thought through this from 3:45 to 4:30 and took some notes. Then I tried to go back to sleep, but unfortunately, that didn\u0026rsquo;t work out. :P\nThe goal in modeling this API as defaults + overrides is gaining the ability to recategorize properties in the future between shared and specific to a service type.\nData Service Notes Record leaf(props...) Record root(List\u0026lt;leaf+string serviceType\u0026gt; serviceTypes, props...) extends leaf map(leaf, root) { prop = coalesce(leaf, root) return transform(prop) } root: ProfileConfig leaf: ProfileServiceTypeConfig Reddit link on why Java Records are not extensible: The restrictions on records may seem arbitrary at first, but they all stem from a common goal, which we can summarize as \u0026ldquo;records are the state, the whole state, and nothing but the state.\u0026rdquo; Specifically, we want equality of records to be derived from the entirety of the state declared in the state description, and nothing else. Were mutable fields, or extra fields, or superclasses permitted, these would each introduce situations where equality of records either ignored certain state components (it is questionable to include mutable components in equality calculations), or was dependent on additional state that is not part of the state description (such as additional instance fields or superclass state). This would have greatly complicated the feature (since surely developers would demand the ability to separately specify which components are part of the equality calculation), and also undermine desirable semantic invariants (such as: extracting the state and constructing a new record from the resulting values should result in a record equal to the original).\nPivot to more static, less dynamic:\nIt will probably be easier to validate required fields through annotations if they are statically either in root or leaf. The same 400 status code could be returned with a custom exception during data mapping, but then lose the automatic spring boot validation root cause analysis in RFC ProblemDetail error returned.\nIt's still an easy mapping change to move a property from one to the other. Also solves the read structure (knowing which properties to include at the root vs. leaf level) without reflection.\nRest of the Morning Bible reading Pushups and dumbbell rows listening to Morse code ( CRSnet streams ) Yogurt and nuts listening to the All-In podcast Shower and developing the idea for this blog post Writing blog post Planning on hitting Chick-fil-A for a chicken biscuit for second breakfast on the way in to work. ","permalink":"https://blog.danieladamstech.com/2023/hammock-driven-development/","summary":"\u0026hellip;Or how good ideas sometimes come in the middle of the night.\nAn episode of the Big Ideas in App Architecture podcast introduced me to a set of talks by Rich Hickey. I think it was this one with Joe Lynch? Rich\u0026rsquo;s talks were Simple Made Easy and Hammock Driven Development . Both were insightful regarding what mindset to invoke when creating software.\nThe premise of hammock driven development is that thinking about a topic over a long enough period of linear time causes the subconscious mind to prioritize it.","title":"Hammock Driven Development"},{"content":"Head over to the search page to try out the newly added search feature!\nHugo has a docs page on search options. Most solutions leverage lunr or fuse, both JavaScript client-side libraries. This theme, Papermod, supports search with fuse.js via Hugo\u0026rsquo;s native JSON output format. When enabled JSON output, I noticed file index.json had the full-text content of all blog posts. Fuse.js pulls this JSON client side and performs a fuzzy search on the content. Right now, the file size is 100KB, not bad. It did make me think about how long it will take to grow large enough that I\u0026rsquo;d rather not have taking up bandwidth. The search page is probably more likely to be used from a laptop where data transfer matters less than on a mobile device.\nPagefind is a static site client-side search library focused on bandwidth efficiency. It released 1.0.0 last week. When I ran the CLI on my site, it generated a binary index for more compression. Playing with it on the Pagefind website leads me to believe it stores compressed full text of the pages instead of just wordlists since it will show sentence fragments the search term was found in. Very interesting!\nAs we all have heard, \u0026ldquo;premature optimization is the root of all evil.\u0026rdquo; The theme\u0026rsquo;s default search on the full JSON dump of all blog posts is a fine place to start. Later if I write enough that it grows to 1MB, I will probably start looking at alternatives. I could investigate the level of compression something like Pagefind gives. It might scale closer to O(log n) and give more of a size difference than it does on my current small site size. Another option would be a search service like Algolia. ","permalink":"https://blog.danieladamstech.com/2023/blog-search-and-premature-optimization/","summary":"Head over to the search page to try out the newly added search feature!\nHugo has a docs page on search options. Most solutions leverage lunr or fuse, both JavaScript client-side libraries. This theme, Papermod, supports search with fuse.js via Hugo\u0026rsquo;s native JSON output format. When enabled JSON output, I noticed file index.json had the full-text content of all blog posts. Fuse.js pulls this JSON client side and performs a fuzzy search on the content.","title":"Blog Search and Premature Optimization"},{"content":"Kishore Koduri: \u0026ldquo;Autonomy, Mastery, Purpose\u0026rdquo; Will Larson: \u0026ldquo;Company, Team, Self\u0026rdquo;\nI find analyzing decision-making values systems thought-provoking. Most times they are orthogonal to each other and tend to give a fuller picture when taken together. The interesting point with this set is the value of Autonomy. Autonomy to do what? Autonomy is closely related to trust. When someone gives you trust, you must steward it appropriately. In several Staff+ engineering books I\u0026rsquo;ve read, a common theme is that with increased autonomy comes increased responsibility for alignment.\nThis is why pairing with Will Larson\u0026rsquo;s set of values brings increased guidance. We should use autonomy to optimize for the company, team, and one\u0026rsquo;s self in that order. He does give the caveat with this framework that being too aggressive can lead to burnout. We can make concessions on small, low-impact projects to optimize more for self for the purpose of maintaining engagement. The company vs. team aspect is also pertinent. We need to evaluate the solutions we\u0026rsquo;re putting in for their support impact on other teams. Stream-aligned teams (to quote Team Topologies) should try to keep things self-contained where possible, and platform teams should focus on the smoothest handoff to their consuming teams.\nFrom Big Ideas in App Architecture Podcast episode and Will Larson\u0026rsquo;s blog respectively.\n","permalink":"https://blog.danieladamstech.com/2023/decision-making-value-systems/","summary":"Kishore Koduri: \u0026ldquo;Autonomy, Mastery, Purpose\u0026rdquo; Will Larson: \u0026ldquo;Company, Team, Self\u0026rdquo;\nI find analyzing decision-making values systems thought-provoking. Most times they are orthogonal to each other and tend to give a fuller picture when taken together. The interesting point with this set is the value of Autonomy. Autonomy to do what? Autonomy is closely related to trust. When someone gives you trust, you must steward it appropriately. In several Staff+ engineering books I\u0026rsquo;ve read, a common theme is that with increased autonomy comes increased responsibility for alignment.","title":"Decision-Making Value Systems"},{"content":"Last weekend I was painting the boys\u0026rsquo; room and signed up for an Audible trial for some longer-form content than my typical podcasts. I\u0026rsquo;m not the fastest painter, clocking about 5 hours for cleaning, taping, and clean up and 6 for painting. By the end, the room looked great and the boys were very happy. As mentioned in a previous post, I\u0026rsquo;ll listen to Will Larson\u0026rsquo;s StaffEng podcast while driving to work. Those interviews have been so informative that I selected to listen to his synthesized book Staff Engineer: Leadership beyond the management track. Will starts the book by categorizing staff+ engineers he\u0026rsquo;s interviewed into the following buckets by behavior.\nStaff Engineer Archetypes Tech Lead: Drives 1-2 team\u0026rsquo;s planning and execution, unblocking along the way as needed via mentoring. This is the most common variety since it occurs roughly at 1 per 8 engineers. Architect: Leads direction, quality, and approach within a specific critical domain. See Brian Chambers Theory of Waves. Solver: Digs deep into arbitrarily complex problems and finds a path forward. Might bounce around the organization as needed. Right Hand (can\u0026rsquo;t resist the Hamilton reference ): Provides additional bandwidth to leaders of organizations by acting on their behalf. Requires tight alignment. What do staff engineers do? Focus areas include mentoring, building their technical brand, and advocating for burning down tech debt while balancing delivering business value. One theme in various leveling rubrics is the time horizon of assignments gets longer at the higher levels. Feedback cycles also slow down commensurately, so the engineer must have good judgment and the ability to glean non-obvious cues to inform adjustments to trajectory. There are benefits to being at a Staff+ level. The title allows you to stop wasting effort proving your seniority. Interviews found this more common with women and minorities. You are also more likely to be allowed access to \u0026ldquo;the room where it happens.\u0026rdquo; There\u0026rsquo;s also increased agency to pick projects, but that comes with higher accountability for alignment of impact from those projects. Don\u0026rsquo;t get too comfortable working on easy and low impact instead of going for hard and high impact. There is a subset of this low-impact work that is high visibility. Too much optimization for visibility can also be a pitfall. \u0026ldquo;Visibility is a transient currency. Learning and developing yourself is a permanent one; focus on the latter once you\u0026rsquo;ve done the minimum to clear the former\u0026rsquo;s cliff.\u0026rdquo; ( quote )\nEffecting change is a primary goal of a Staff engineer. The book contained a full section on methods for driving improvement in technical quality. One principle was to start with small, concrete changes that can layer on each other rather than rolling out a multifaceted program all at once. Engineers will be able to digest and internalize the slow, steady pace of change more effectively than split attention would allow. Another benefit of stepwise change is it allows monitoring the target metrics along the way to understand when a change moved the needle in a positive direction versus non-significant change or regression. Interesting but telling quip: \u0026ldquo;It doesn\u0026rsquo;t reduce risk, just makes it more clear who to blame when things go wrong.\u0026rdquo; Beyond audit requirements, we must make sure the processes we are creating actually reduce risk and not just service to point out problems after the fact.\nCreate space for others. The way a Staff engineer can be most valuable to his or her employer is to have a multiplicative effect on others. In order to let others flourish and solve problems on their own, you need to not be involved in everything. I have made the mistake in the past of being involved in too many decisions that it became hard to pull back. Make your feedback non-blocking on a document or code review (i.e. label the comment as a \u0026ldquo;nit\u0026rdquo;). This lets people assimilate what you\u0026rsquo;re saying while still exercising their own power of decision and building that muscle. When you do make a decision, write down the thinking process and research paths that lead to that decision. Even supporting arguments for the decision do not bring others along in the reasoning process as well as laying out the various paths of your research. This still needs to be balanced and well-organized to keep the reader on track.\nOrganizational authority is loaned, not given. It can be retracted. Using it depends on staying aligned with the sponsor that gave it to you. This idea of stewarding the responsibility you were given also applies to \u0026ldquo;remaining in the room.\u0026rdquo; A character quality that applies to both async decision-making (RFPs, code reviews) as well as in-person interactions is a low-friction demeanor. Someone used their social capital to invite you into the room. You should act in a way that makes the other members respect your sponsor\u0026rsquo;s judgment call of involving you. One of the most important things (especially for opinionated people such as myself) is not sucking the oxygen out of the room. No one else wants to work in that environment. Don\u0026rsquo;t act like every small decision is \u0026ldquo;threat level 1000\u0026rdquo; since, in reality, most are not. The second most important takeaway from this section was about being understood. I love this quote: \u0026ldquo;Keep in mind that it\u0026rsquo;s your obligation to be understood, not the obligation of everyone else to understand you.\u0026rdquo; It is similar to the idea in the Bible how Jesus came to serve rather than be served. Will includes a chapter in the book on presenting to executives. My only tangential experience was helping plan a Data Modernization report-out some managers were giving, so I found this insight fascinating. The book says that executives are used to and very adept at processing data presented in a particular way. Your presentation will be most effective if you align with that methodology. For example, the executive might ask questions for pattern matching or expect data points up front rather than saving for the end. Get a clear understanding of why you are presenting and what you hope to accomplish: planning, status reporting, or resolving misalignment. Your goal is to extract their perspective and understand how to align with their priorities, not causing them to change their mind. Structured writing is important to effectively prepare thoughts, see The Pyramid Principle by Barbara Minto. Below are the 4 points in the SCQA format. I plan to start utilizing this in my writing.\nSCQA Format Situation: Provide context. Complication: Why is the current situation problematic? Question: State the question succinctly. Answer: Propose your solution. Concise writing and speaking lets you contribute more ideas in less time. Many times your opening paragraph in SCQA format is all that is needed to start the conversation. However, the planning necessary to write the full document will pay dividends when speaking in person. After you\u0026rsquo;ve written the document, ask for review by peers and stakeholders. The first round will be refining concepts and communication, so balance the seniority of those reviewing. You want to receive the necessary feedback without using up your time allotment from those you want to be engaged at approval time. The second round is with those decision-makers to get alignment prior to the meeting. That will help ensure as much as possible that there are no surprises at the meeting and the official approval is almost a formality. This is exactly the function of the majority whip in Congress. They ensure the necessary support is committed before calling for a vote.\nHere are several link-outs from the book for my followup reading list. They were suggested as ways to dive deeper into the specific topics.\nBeing Glue Mentorship and Sponsorship A Framework for Responsible Innovation How Big Technical Changes Happen at Slack Building Evolutionary Architectures In a similar vein, on Tuesday afternoon I watched a Staff Engineering panel on Zoom. It was super inspiring to hear what some of the people were building. They were also very close to my age, younger than I was expecting for people so senior. One resource I would suggest following is Ryan Peterman\u0026rsquo;s newsletter developing.dev . It is short yet hits on important topics.\n","permalink":"https://blog.danieladamstech.com/2023/staff-engineer-by-will-larson/","summary":"Last weekend I was painting the boys\u0026rsquo; room and signed up for an Audible trial for some longer-form content than my typical podcasts. I\u0026rsquo;m not the fastest painter, clocking about 5 hours for cleaning, taping, and clean up and 6 for painting. By the end, the room looked great and the boys were very happy. As mentioned in a previous post, I\u0026rsquo;ll listen to Will Larson\u0026rsquo;s StaffEng podcast while driving to work.","title":"Staff Engineer by Will Larson"},{"content":"Back at the start of 2021, I delved into a small curiosity project around how to assert ownership of an AWS role to a non-AWS entity. I implemented an API Gateway Sigv4 signer in a Spring RestTemplate Interceptor . Later we integrated that design in a production app. That security integration has had zero issues since. As a thought experiment, I wanted to see if it was possible to use an IAM root of trust when calling other endpoints besides API Gateway. AWS Sigv4 authentication is a symmetric scheme, but to keep things simple with fewer security resources on the verification side, I knew I needed to use asymmetric verification.\nThis idea was inspired by CyberArk DAP IAM Authenticator integration and Hashicorp Vault IAM authenticator. Vault has a lot of cool technology in it! For another side project, I tried combining the Shamir Secret Sharing package from Vault with the Format-Preserving Encryption package from CapitalOne into my project shamirfpe .\nBack to the topic at hand. The client wanting to assert ownership of an IAM role will pre-sign an STS GetCallerIdentity request, which can then be executed by a trusted entity. Handing over this signature will not compromise the security of the IAM role secret keys. The trusted entity will get a response directly from AWS STS, which can be trusted. We then know the caller has that specific role session. Even administrators will not be able to generate application tokens if they are not trusted in the assume role policy. This maintains the same security posture as IAM-based access to AWS resources.\nThe presign and execute process could be done on every app request, but making a network call every time is slow and susceptible to throttling. We instead sign that assertion with KMS asymmetric keys . The RSASSA_PKCS1_V1_5_SHA_256 algorithm supported by KMS is the same one used for RS256 signed JWTs. Constructing the appropriate JWT header and body allows us to issue industry-standard tokens that can be verified by client libraries across many languages. In the standard JWT verification process, the verifier can have multiple public key IDs to support rotation of a specific key upon compromise. The backup signing key[s] can have deny policies in KMS during normal operation. If a key is compromised, it can be scheduled for deletion thereby disabling the key. The next key can be enabled for usage without applications having to always check the public key endpoint.\nBelow is a sequence diagram representing the three parties (client, assertion validator, and target resource) and two AWS services (STS and KMS.)\nMy previous Sigv4 signer integration was in Java using the AWS SDK AWS4Signer , so I wanted to use some other languages for this exercise.\nRequest formulation and presigning JavaScript via AWS SDK middleware . This is a more roundabout hack method, but the SDK manages pulling credentials from the metadata endpoint and maintaining a fresh session token component JavaScript via AWS SDK SignatureV4 module. The code is very straightforward, but doesn\u0026rsquo;t pull the credentials for you. The cleanest way is using the external dependency from Michael Hart aws4 Posting to STS, signing with KMS, and constructing JWT - Python JWT verification - Go The main shortcoming of this idea and why it isn\u0026rsquo;t practical for use by itself in a production scenario is it only addresses authentication, not authorization. I believe AWS IAM roles are about the best root of trust available, but a good authorization system is necessary. Hashicorp Vault and CyberArk bring robust implementations on the authorization side, but that is not something I would want to code custom.\nThis is where AWS VPC Lattice comes in. I was very excited to hear about it when I attended Re:Invent last year. I switched my schedule around to go to a talk by the Lattice team. AWS API Gateway private APIs to backends over PrivateLink accomplishes something similar from a security perspective but has a much more complex set of AWS resources. That is worth it when you are trying to achieve API governance, but sometimes you just want a simple connection with good security. The VPC lattice target group construct achieves this perfectly. VPC lattice uses IAM-format auth policies , very similar to API Gateway IAM authentication. I was hoping that the ARN format would be a stable identifier based on service name so they would still work if a service was deleted and recreated, but it seems to be a random ID like API Gateway. I looking forward to reading more about and hopefully playing with VPC Lattice in the future.\n","permalink":"https://blog.danieladamstech.com/2023/iam-assert-role/","summary":"Back at the start of 2021, I delved into a small curiosity project around how to assert ownership of an AWS role to a non-AWS entity. I implemented an API Gateway Sigv4 signer in a Spring RestTemplate Interceptor . Later we integrated that design in a production app. That security integration has had zero issues since. As a thought experiment, I wanted to see if it was possible to use an IAM root of trust when calling other endpoints besides API Gateway.","title":"IAM Assert Role"},{"content":"Cloud Service Dependencies To quote Werner Vogles, \u0026ldquo;Everything fails all the time.\u0026rdquo; When designing an app, we want to carefully evaluate what dependencies it requires. Cloud services are highly available, but the union of many can still lead to a measurable decrease in availability. Dependencies that are not absolutely necessary should fail open to allow the system to continue doing critical work. Below are two examples I have personally run into in the past couple of years.\nCaching Proxy DynamoDB Write We had a web app with the UI layer running out in AWS that used an authentication system on-prem. Session validation on every page load and Ajax call was causing performance issues, so I wrote a session caching service out in AWS to decrease the latency for validation. It was a huge success for the user experience, saving over 15 hours of waiting time per day. The service used a two-tier cache system: local cache within the service and DynamoDB within the cluster. Cache misses called the on-prem system.\nThe December 7th, 2021 issue caused a DynamoDB outage for us. According to the postmortem, using VPC endpoints to connect to DynamoDB was what caused the impact. The service was correctly set to opportunistically write the response back to DynamoDB. The issue was my lookup only caught my custom TokenNotFoundException. DynamoDB failures threw a different exception which my code was not expecting. I made a quick fix to go to origin on any failure, but we ended up waiting out the outage and not trying to deploy during it. Only a portion of users were affected, and we didn\u0026rsquo;t want to make the situation worse by the ongoing outage causing a deployment issue. After the outage and much testing with simulated conditions like removing IAM access to the table, we deployed the change to production.\nDocker Log Driver Ring Buffer One issue that hit us out of left field was the Kinesis outage of November 25th, 2020. We had on-prem and ECS Docker services that froze up and were unable to deploy new. None of our applications used Kinesis, so what was going on? It turned out that the Docker log drivers can actually block the applications if logs aren\u0026rsquo;t accepted. Cloudwatch logs depended on Kinesis and therefore couldn\u0026rsquo;t ingest logs. I was surprised this was default behavior so opened a Github issue asking if there was any plan to change. We set all our services to non-blocking with a buffer after that event. AWS later released a blog post addressing this issue and presenting the options tradeoffs.\nApplication Service Dependencies Which cloud services you use might be a bit more in your purview, but sometimes what other application services you interact with (especially if owned by other teams) can be necessitated by the requirements. We still should take explicit inventory of those dependencies. Having that information will allow us to better share knowledge about the system as well as prepare us for responding to incidents. If a downstream service encounters an outage, we can know what to expect from our system.\nKnowing dependency ordering to bring up the whole system from a hard-down outage is important. Figuring that out before an incident will be much easier than a chat storm during. Extra care must be taken when trying to identify circular dependencies that can inhibit the application from coming up successfully. A recent interesting read along those lines is Gergely Orosz\u0026rsquo;s article Inside Datadog’s $5M Outage. Many times these circular dependencies are at the control plane level. This is why we try to use services with managed control planes instead of running those ourselves. However, circular dependencies can still exist within applications, so we must be careful with those as well.\nHealth Check Dependencies Last week, I saw a service in Dev that was continually flapping unable to come up healthy. After looking at the ECS statuses and logs, it became apparent that the health check was failing, and the health check was failing due to a synthetic DynamoDB read failing. That health check was looking up a particular test data record that was no longer there. The dev database had probably been cleared at some point to start over fresh which caused the service to start flapping.\nHealth checks should depend on another service (cloud or app) only if the only way to recover from that service being down is to restart your container. This should almost never be the case as a service should be able to reinitialize connections to its dependencies without a full restart.\nThe AWS Builders Library has a great article about things to watch for when implementing health checks . Two choice quotes below:\nDependency health checks are appealing because they act as a thorough test of a server’s health. Unfortunately they can be dangerous because a dependency can cause a cascading failure throughout a system.\nThe difficulty with health checks is this tension between, on the one hand, the benefits of thorough health checks and quickly mitigating single-server failures and, on the other hand, the harm done by a false positive failure across the entire fleet. Thus, one of the challenges of building a good health check is to guard carefully against false positives. In general, this means that the automation surrounding health checks should stop directing traffic to a single bad server but keep allowing traffic if the entire fleet appears to be having trouble.\nHopefully this post has given some food for thought about being cognizant of system dependencies, criticality of them, and plans for when they fail.\n","permalink":"https://blog.danieladamstech.com/2023/system-dependencies/","summary":"Cloud Service Dependencies To quote Werner Vogles, \u0026ldquo;Everything fails all the time.\u0026rdquo; When designing an app, we want to carefully evaluate what dependencies it requires. Cloud services are highly available, but the union of many can still lead to a measurable decrease in availability. Dependencies that are not absolutely necessary should fail open to allow the system to continue doing critical work. Below are two examples I have personally run into in the past couple of years.","title":"System Dependencies"},{"content":"My Uncle Stephen encouraged me to read Theodore Roosevelt\u0026rsquo;s The Man In The Arena last summer. It has been in the back of my mind until I heard it mentioned again on a podcast alongside a list of influential biographies. I looked it up and found it was readable in just one night, so I pulled a \u0026ldquo;carpe noctem\u0026rdquo; and finished it. These quotes stand on their own, so my additions will be minimal.\nOne must try in order to truly live.\nIt is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows the great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.\nDemocratic republics put the onus on all citizens to live as they want the greater country to function.\nThe average citizen must be a good citizen if our republics are to succeed. The stream will not permanently rise higher than the main source; and the main source of national power and national greatness is found in the average citizenship of the nation. Therefore it behooves us to do our best to see that the standard of the average citizen is kept high; and the average cannot be kept high unless the standard of the leaders is very much higher.\nSound character \u0026gt; sound mind \u0026gt; sound body \u0026gt; scholarship.\nInternal control \u0026gt; external control.\nLet those who have, keep, let those who have not, strive to attain, a high standard of cultivation and scholarship. Yet let us remember that these stand second to certain other things. There is need of a sound body, and even more of a sound mind. But above mind and above body stands character—the sum of those qualities which we mean when we speak of a man’s force and courage, of his good faith and sense of honor. I believe in exercise for the body, always provided that we keep in mind that physical development is a means and not an end. I believe, of course, in giving to all the people a good education. But the education must contain much besides book-learning in order to be really good. We must ever remember that no keenness and subtleness of intellect, no polish, no cleverness, in any way make up for the lack of the great solid qualities. Self-restraint, self-mastery, common sense, the power of accepting individual responsibility and yet of acting in conjunction with others, courage and resolution—these are the qualities which mark a masterful people. Without them no people can control itself, or save itself from being controlled from the outside.\nAdmire the character, actions, and journey. Not merely the end state of success.\nNevertheless, while laying all stress on this point, while not merely acknowledging but insisting upon the fact that there must be a basis of material well-being for the individual as for the nation, let us with equal emphasis insist that this material well-being represents nothing but the foundation, and the foundation, though indispensable, is worthless unless upon it is raised the superstructure of a higher life. That is why I decline to recognize the mere multimillionaire, the man of mere wealth, as an asset of value to any country; and especially as not an asset to my own country. If he has earned or uses his wealth in a way that makes him a real benefit, of real use—and such is often the case—why, then he does become an asset of real worth.\nBut it is the way in which it has been earned or used, and not the mere fact of wealth, that entitles him to the credit. There is need in business, as in most other forms of human activity, of the great guiding intelligences. Their places cannot be supplied by any number of lesser intelligences. It is a good thing that they should have ample recognition, ample reward. But we must not transfer our admiration to the reward instead of the deed rewarded; and if what should be the reward exists without the service having been rendered, then admiration will only come from those who are mean of soul.\nIdealistic visions can be the detriment of practical progress. This holds true in software software engineering.\nThe citizen must have high ideals, and yet he must be able to achieve them in practical fashion. No permanent good comes from aspirations so lofty that they have grown fantastic and have become impossible and indeed undesirable to realize. The impractical visionary is far less often the guide and precursor than he is the embittered foe of the real reformer, of the man who, with stumblings and shortcomings, yet does in some shape, in practical fashion, give effect to the hopes and desires of those who strive for better things. Woe to the empty phrase-maker, to the empty idealist, who, instead of making ready the ground for the man of action, turns against him when he appears and hampers him as he does the work!\nDon\u0026rsquo;t support someone who promises to elevate your group to the detriment of another.\nOf one man in especial, beyond any one else, the citizens of a republic should beware, and that is of the man who appeals to them to support him on the ground that he is hostile to other citizens of the republic, that he will secure for those who elect him, in one shape or another, profit at the expense of other citizens of the republic. It makes no difference whether he appeals to class hatred or class interest, to religious or anti-religious prejudice. The man who makes such an appeal should always be presumed to make it for the sake of furthering his own interest. The very last thing an intelligent and self-respecting member of a democratic community should do is to reward any public man because that public man says that he will get the private citizen something to which this private citizen is not entitled, or will gratify some emotion or animosity which this private citizen ought not to possess.\n","permalink":"https://blog.danieladamstech.com/2023/the-man-in-the-arena/","summary":"My Uncle Stephen encouraged me to read Theodore Roosevelt\u0026rsquo;s The Man In The Arena last summer. It has been in the back of my mind until I heard it mentioned again on a podcast alongside a list of influential biographies. I looked it up and found it was readable in just one night, so I pulled a \u0026ldquo;carpe noctem\u0026rdquo; and finished it. These quotes stand on their own, so my additions will be minimal.","title":"The Man in the Arena"},{"content":"Lately, I have been fascinated with researching the behaviors of Staff Engineers. They focus on having the highest possible impact on the organization. Generally, this means enabling and leveling up others around them over just concentrating on their personal contributions. As a Senior engineer, I still do a lot of designing and implementing instead of being primarily others-focused. However, I still try to pick up these behaviors and use them on the side when fitting.\nResources Books The Staff Engineer\u0026rsquo;s Path - Tanya Reilly Staff Engineer: Leadership beyond the management track - Will Larson (This is on my to-read list, just listening to the interviews podcast the book is based on now.) Other books Designing Data Intensive Systems - Martin Kleppmann , a very good book and jumping-off place for further research. I should re-read it sometime and write a post. Team Topologies - Matthew Skelton and Manuel Pais, my notes blog post The Software Engineer\u0026rsquo;s Guidebook - Gergely Orosz, coming Fall 2023. This isn\u0026rsquo;t solely about Staff level, but the progression from fundamentals to Distinguished Podcasts StaffEng - Will Larson, interviews backing the StaffEng book Big Ideas in App Architecture - the host David Joy is uplifting to listen to, such a genuinely positive/invested/empathetic/grateful person. Also great tech discussions! Software Engineering Daily - hit or miss, a lot of times it is heavy on the product push even if the engineering content is there The Changelog - laid-back, easy listening Websites and Personal Blogs StaffEng - Will Larson The Pragmatic Engineer - Gergely Orosz Chamber of Tech Secrets - Brian Chambers Matt Rickard\u0026rsquo;s blog Corporate Tech Blogs (The ones I subscribe to on Feedly. I need more good ones, thinking of dropping the Product blogs.)\nSpotify Engineering CloudFlare blog Chick-fil-a Tech Netflix Tech Blog Product blogs: AWS and Hashicorp ","permalink":"https://blog.danieladamstech.com/2023/staff-engineer-resources/","summary":"Lately, I have been fascinated with researching the behaviors of Staff Engineers. They focus on having the highest possible impact on the organization. Generally, this means enabling and leveling up others around them over just concentrating on their personal contributions. As a Senior engineer, I still do a lot of designing and implementing instead of being primarily others-focused. However, I still try to pick up these behaviors and use them on the side when fitting.","title":"Staff Engineer Resources"},{"content":"The Prime Video team made some waves back in March when they wrote about refactoring their distributed microservices application into a monolith . It was a topic of various Medium and LinkedIn posts. Then two weeks later Werner Vogels published a follow-up post Monoliths are not Dinosaurs . Generally, Werner is presenting about microservices and distributed event-driven architectures at Re:Invent, but he seemed to step in here to say engineering judgment is important and microservices are not always the right decision. At scale, actually profiling the workload is important. Converting media is a CPU-intensive process, but through empirical testing, the team found it was cheaper to transcode more than once than to pass around the files via S3.\nWerner used another Prime video post about distributed work scheduling and prioritization as a contrast to the monolithic architecture. This one implemented a serverless, event-based architecture in a more economical way. Big-O notation doesn\u0026rsquo;t exactly apply here, but it could be slanted to say the video publishing queuing system was O(1) since it ran through the Lambda + SQS portion of the system once per job. On the other hand, the video quality analysis system \u0026ldquo;performed multiple state transitions for every second of the stream\u0026rdquo; which could be considered O(n) as it was constantly spinning throughout the duration of the job. This looping could have been what contributed to the S3 cost as the iteration speed of the Step Function seems to indicate repeatedly passing around chunks of video for real-time monitoring.\nMy Analysis I\u0026rsquo;ll start by saying that I\u0026rsquo;m a big fan of Step Functions for certain use cases. I recently built two projects in Step Functions for our Data Modernization control plane. One was an ML Pipeline with user-submitted code hooks to dynamically select phases at runtime . The other was an event-based ETL run manager. Files would arrive at S3, Eventbridge Pipes would route those S3 events to SQS queues, and a StepFunction would run the ETL on queue depth alarm and every ten minutes thereafter until the queue was cleared. These have been low-cost but high-impact processes.\nStep Functions is very low maintenance - no language runtime or dependency upgrades. However, it seems like when writing them, I had to plan around limitations of the intrinsic functions and more limited manners that you could merge state. For example, there is no function to test if a value exists in an array.\nKnow your \u0026ldquo;why\u0026rdquo; Step Functions can be used for many purposes. Knowing for what reason you chose the service can help inform how much you are willing to pay.\nStatus tracking and debuggability of important processes \u0026ldquo;Glue\u0026rdquo; orchestrating disparate systems Replacing custom orchestration code for reduced maintenance Use for things that \u0026ldquo;matter\u0026rdquo; You have now figured out which benefits you wish to reap from centralized workflow management with Step Functions. The next step is evaluating how much those matter to the use case at hand. The top benefit in many of my scenarios is the innate status tracking for each transaction. The same detail can be gleaned from custom instrumentation or monitoring tools, but Step Functions naturally tracks any process whether sync or async. This enhanced tracking could be useful for things like financial transactions or delivering critical files to business partners. Most people would happily spend a couple more dollars on the AWS bill for peace of mind around a solid workflow engine running those processes.\nIn the Prime Video use case, tracking and debuggability are not that important for a monitoring tool. The team probably chose the solution for other reasons when it was first implemented. Along this axis, a less-natively-traceable solution like SQS queues would still fit the bill. With decentralized orchestration like SQS, passing things like OpenTelemetry trace headers as SQS message attributes becomes more important for operational monitoring.\nPay attention to cost per business process execution Straight-through processes with constant O(1) costs are easy to estimate. O(n) processes with looping can be estimated in the average case, but the high iteration case needs accounted for as well. Many times these loops are for polling \u0026ldquo;legacy\u0026rdquo; processes that are not event-based. Switching to an Activity task type can remove iteration from the StepFunction. If that is not possible, using exponential backoff and a timeout exit after a maximum number of polls can make the maximum invocation cost easier to predict.\nIf the cost/benefit is borderline, try to gauge the scaling pattern of the workload over the next couple years Werner says in his blog post:\nMy rule of thumb has been that with every order of magnitude of growth you should revisit your architecture, and determine whether it can still support the next order level of growth.\nPremature optimization is never a good thing. However, looking ahead at projected growth can give a sense of the timeline a solution can stay in place. In a high-growth app, you could easily hit an order of magnitude in two years and be forced to iterate to the next design. In my industry of mortgage insurance, our business is tied to the housing market. Some areas of the business like orders are tied to that fixed-size market. Others like data analytics will probably see a 100x growth over the next ten years. For many of our transactional areas, we can design for long-term maintainability and don\u0026rsquo;t need to redesign for scale every two years. If the economics don\u0026rsquo;t look promising in the long term, then it could be good to consider the options that would be less of a drastic change to the system as it evolves over time.\n","permalink":"https://blog.danieladamstech.com/2023/step-functions-use-cases/","summary":"The Prime Video team made some waves back in March when they wrote about refactoring their distributed microservices application into a monolith . It was a topic of various Medium and LinkedIn posts. Then two weeks later Werner Vogels published a follow-up post Monoliths are not Dinosaurs . Generally, Werner is presenting about microservices and distributed event-driven architectures at Re:Invent, but he seemed to step in here to say engineering judgment is important and microservices are not always the right decision.","title":"Step Functions Use Cases"},{"content":"CAP Theorem Recently I listened to CAP Theorem 23 Years Later with Eric Brewer on the Software Engineering Daily podcast. I didn\u0026rsquo;t recognize the name at first (shame on me), but it was very cool to hear about the CAP theorem from the original postulator.\nI haven\u0026rsquo;t had a chance to work on distributed systems in real life, but I find the topic fascinating. Coordinated cache eviction was the closest I\u0026rsquo;ve come. I read the book Designing Data-Intensive Applications soon after it was released. Besides the in-depth content within, Martin Kleppmann includes so many footnotes in that book it is the perfect jumping-off point into any other tangent of databases, data processing, or distributed systems. I got halfway through his undergrad Distributed Systems lecture series with lecture notes he taught at Cambridge. I completely forgot about that course until writing this blog - definitely will revisit it soon.\nAnyway, back to the CAP theorem . It says that in a distributed system, only two of Consistency, Availability, and Partition Tolerance can be achieved. We take partitions as a fact of life (cables cut, power outages, etc) so we are only left with trading off Consistency and Availability. This makes logical sense at a basic level. If we have a cluster that has split down the middle and one side gets a write, that write will not be visible on the other side. The system has a choice to make at this point. Don\u0026rsquo;t give any answer since it could be wrong (Consistency), or give the answer it has (Availability). Different angles on these properties exist, a common one being read availability vs write availability.\nAs Eric Brewer highlights on the podcast, the CAP theorem can be taken too black-and-white. There are opportunities to find niches of acceptable tradeoffs based on the desired goals. For example, Google Cloud Spanner uses Google\u0026rsquo;s TrueTime API to expose time uncertainty. The Spanner Paper (2012) explains how the database adapts commit timing based on how in-sync the clocks are to preserve transaction ordering. This allows Spanner to provide the highest levels of consistency . While driving to a 30th birthday weekend campout, I listened to an interview with Peter Mattis , co-founder and CTO of Cockroach Labs. I knew Cockroach DB had some distributed systems properties, but I didn\u0026rsquo;t realize the VC pitch when starting the company was some ex-Googlers \u0026ldquo;implementing the Spanner paper.\u0026rdquo;\nFree Will and Determinism On an in-one-sense parallel topic, I listened to the London Lyceum podcast on Free Will and Determinism . I was somewhat familiar with the two concepts but had never heard of Compatibilism and Incompatibilism . These distinctions reflect whether one can hold to both Free Will and Determinism at the same time - if they are logically compatible. As this was my first time hearing about those philosophies, I would need to do more research before commenting further.\nGray Area Last weekend during my birthday campout, we completed a nine-mile hike Friday, four-mile hike Saturday, and two Summits on the Air . After all that, we had a four hour campfire Saturday night. It was a great time of relaxing and reflecting. One topic we talked about was how narrow our knowledge is and how we must lean on others for the rest. My friend has a master\u0026rsquo;s in early American history, a J.D., and practices law in Virginia. I work in IT around cloud, data, and most recently software engineering. At the end of the day, these are very narrow slices of knowledge. Even topics I am interested in like distributed systems I can\u0026rsquo;t talk intelligently about at a deep level. Much less philosophy. 😄\nBoth CAP Theorem and Free Will vs Determinism seem to contain opposing positions to the casual observer. A database can be either available OR consistent. A person can subscribe to either Free Will OR Determinism. But in reality, the subject is much more nuanced. People that research the area can navigate the minutiae and find useful ground in the middle. It benefits everyone to have those people that can dive in deeply and make discoveries for the rest of us. Distributed databases would be far less useful if we threw up our hands and accepted only fully consistent or fully available. For consumers of the technology or philosophical thought, we need to make a conscious decision on how much understanding we need to reap the benefits we are aiming for. While fuller understanding makes our lives richer and our minds more appreciative of others\u0026rsquo; work, the beauty of abstractions is we don\u0026rsquo;t need to be an expert to partake.\n","permalink":"https://blog.danieladamstech.com/2023/cap-theorem-free-will-and-determinism/","summary":"CAP Theorem Recently I listened to CAP Theorem 23 Years Later with Eric Brewer on the Software Engineering Daily podcast. I didn\u0026rsquo;t recognize the name at first (shame on me), but it was very cool to hear about the CAP theorem from the original postulator.\nI haven\u0026rsquo;t had a chance to work on distributed systems in real life, but I find the topic fascinating. Coordinated cache eviction was the closest I\u0026rsquo;ve come.","title":"CAP Theorem, Free Will, and Determinism"},{"content":"I like RSS as an idea/protocol as well as a reading experience. It is one of the few places on the internet where you can consume only desired content and don\u0026rsquo;t have algorithms pushing something else. Pulling RSS feeds from various sites embodies the peer-to-peer nature of the internet. Over time, we have swayed toward central entities being our conduit to information contained on other sites (Google, Facebook, Twitter, link aggregators like Reddit).\nI used Google Reader in high school until it was shut down during my college years. This history article gives a view inside how the \u0026ldquo;project\u0026rdquo; started and how it continually struggled to get recognized as a \u0026ldquo;product\u0026rdquo; within Google.\nTo set up my blog\u0026rsquo;s RSS feed, I first looked in the build directory and saw index.xml . Cool! I typed my site address into Feedly and it did pull a feed. However, the default implementation only uses the .Summary text as part of the RSS feed. This by default is the sentences that include the first 70 words, but can be customized by a \u0026lt;!--more-- \u0026gt; divider. My issue wasn\u0026rsquo;t with the length of the summary, but that it was not clear that there was more to the article. I looked into a way to add a \u0026ldquo;Continue reading at $LINK\u0026rdquo; text at the end. That required changing the template. The respective Hugo framework and Papermod theme RSS templates were very similar and didn\u0026rsquo;t have a provision for tacking on an ending footer. I wanted to stay aligned with upstream so didn\u0026rsquo;t go that route.\nThe Papermod theme adds this provision for full article text. You don\u0026rsquo;t get article page view counts if I added basic, non-invasive analytics in the future, but I liked the full article in the feed better as a reading experience. So I set the show full text parameter in this commit .\n1 2 3 {{- if site.Params.ShowFullTextinRSS }} \u0026lt;content:encoded\u0026gt;{{ (printf \u0026#34;\u0026lt;![CDATA[%s]]\u0026gt;\u0026#34; .Content) | safeHTML }}\u0026lt;/content:encoded\u0026gt; {{- end }} I wanted to make sure the increased size of the full article text wouldn\u0026rsquo;t have a negative impact on feed XML load time so capped it. This pre-processor code at the start of the feed template has a parameter to limit to a certain number of the most recent posts. I set it to 2 for testing and then updated the parameter to a value of 20.\n1 2 3 4 {{- $limit := site.Config.Services.RSS.Limit -}} {{- if ge $limit 1 -}} {{- $pages = $pages | first $limit -}} {{- end -}} Then after making the change, I found the official Hugo RSS docs that mentioned the parameter should be rssLimit. My previous configuration of services.rss.limit must have been an internal parameter, so I updated to the \u0026ldquo;public API surface\u0026rdquo; here . Now, everything is set up as desired!\n","permalink":"https://blog.danieladamstech.com/2023/rss-setup/","summary":"I like RSS as an idea/protocol as well as a reading experience. It is one of the few places on the internet where you can consume only desired content and don\u0026rsquo;t have algorithms pushing something else. Pulling RSS feeds from various sites embodies the peer-to-peer nature of the internet. Over time, we have swayed toward central entities being our conduit to information contained on other sites (Google, Facebook, Twitter, link aggregators like Reddit).","title":"RSS Setup"},{"content":"Both Reddit and Twitter made policy changes over the last couple of months to restrict API access purportedly due to large increases in LLM training traffic.\nCentralized Social Media I was just refreshing myself on the Reddit situation timeline on Wikipedia . Reuters reported February 14 that Reddit was looking to IPO in the second half of 2023. April 18th brought the announcement that Reddit would be charging for API access after a 60-day notice period ( Reddit , The Verge ). Mods staged a unified protest, shutting down several of the top subreddits June 12-14 by making them private. Some subreddits stayed private after the 48-hour \u0026ldquo;blackout.\u0026rdquo; Understandably, Reddit was going to do whatever possible to get back to business-as-usual and keep moving toward IPO. Two days later on June 16th, mods got a message from Reddit basically saying \u0026ldquo;Open up or get replaced.\u0026rdquo; This reminds me of the Israelite captivity by Nebuchadrezzar of Babylon and how he set up Zedekiah as a puppet king. ( 2nd Kings 24:17 ) By the way, that didn\u0026rsquo;t turn out well for him later when he revolted. ( 2nd Kings 25 )\nBased on the investigation of Gergely Orosz of the Pragmatic Engineer, it seems like there was a connection between Twitter\u0026rsquo;s rate limits and cost cutting on GCP ML services . Evidently those services are needed to provide the level of service people have expected from Twitter over the years.\nInstagram has seized the opportunity to launch their Threads product. They have not imposed any limits, probably to draw Reddit and Twitter users towards their platform. One analysis I was listening to (pretty sure from the All In podcast) noted that Facebook and Instagram are both now image and video-heavy platforms with one-way posting. Interactive discussion, like on Reddit and Twitter, provides much better LLM training material. The Threads app data collection policy pointed out by Jack Dorsey highlights this.\nThe [Future?] Fediverse The Fediverse promises a different model of data ownership. The content creators have more control over their data and can take it with them when switching hosts or servers. Mastodon (Twitter replacement), Lemmy (Reddit replacement), and Bluesky with its AT protocol are all apps in this category. It is too early to tell which ones will get long-term, mainstream adoption. Mastadon.help counts over 11,000 instances currently running.\nLet\u0026rsquo;s look at various options for gathering LLM training data from a decentralized system.\nScrape the Fediverse servers In this option, extracting the conversations requires connecting to each server. With over 11,000 Mastodon servers, that sounds like it would take quite the coordination. In reality, people would probably track the number of active users and quality of interactions and just pull from the top 5% of servers. There is probably a long tail of experimental servers that never take off. Even with a small subset of total servers, maintaining the data pipelines seems like a yak shave taking effort away from using the data for model training. It also takes more time and infrastructure to pull historical data as just getting from the current time forward is probably not enough for the training needs.\nBadly secured servers could have the majority of traffic coming from bots instead of real users. What does it take to stop this? Anonymous access is easy to turn off, but the question is how much ML does it take to effectively stop data scraping by crawler accounts. Static validations like captchas or mobile device attestation via Private Access Tokens can prove human user at a point in time. Analytics like access pattern outlier detection can provide ongoing behavior monitoring in case a user is signing up manually on a real device and then selling accounts to a bot network. This could become a big cat-and-mouse game for people running servers.\nHonestly, this route sounds like a lose-lose for the people running the infrastructure and the people trying to train the models. All hosting parties have to worry about bot management, and model trainers have to worry about data pipelines.\nAdd a node on the Fediverse to capture events Organizations that want to train models could hook into the system and get streaming updates. At first glance, this doesn\u0026rsquo;t look very promising. The ActivityPub Follow activity seems like it is Actor specific and would require scraper accounts to follow everyone from whom they wanted to pull data. Mastodon has a \u0026ldquo;local timeline\u0026rdquo; that includes all posts on that server but can\u0026rsquo;t be subscribed to from another server. This is different than blockchain where every node on the chain has a full history. Social media has too much data streaming in to make that model tenable. It seems like option #1 where you must visit all servers and manage those pipelines.\nRely on a company to give you easily queryable, analytics-ready extracts If the fediverse takes off, there will probably be companies that aggregate and sell access to a corpus for model training. This is similar to gateway products like Cloudflare Web3 for Ethereum and IPFS. These products provide live API access to blockchain-based systems. The model training providers would be slightly different in that they are serving optimized, compacted historical (up to the present) data sets. Two methods this data could be surfaced are file-based access and live query capability.\nFile-based sharing is the method taken by Snowflake Marketplace and AWS Data Exchange . Providers solely sharing data and not managing consumer compute vastly simplifies their business. In all, it is probably more cost-efficient to have one consumer compute layer than two compute layers - a query engine on the provider side and processing on the consumer side. Spark and other analytics tools are used to dealing with the S3 API. Spark SQL uses Catalogs to configure connections for querying Iceberg tables natively. Iceberg tables seem to be a popular abstraction for file-based data lakes when not using a more managed data layer like Snowflake. File-based sharing could also be done on one\u0026rsquo;s own without being a part of a data exchange, but it could be a bit more effort to onboard customers than if they have an already established pattern with one of the major data exchanges.\nA query engine approach would probably be similar to Amazon Athena where customers can pay for managed query compute per GB scanned. This requires more multi-tenant infrastructure management on the part of the data provider. It can take a lot of engineering to cover the security and \u0026ldquo;noisy neighbor\u0026rdquo; issues when trying to run multi-tenant query infrastructure. The decision probably comes down to how much flexibly consumers want when consuming the data. File level access will be cheaper, but query can give consumers a more targeted data set.\n","permalink":"https://blog.danieladamstech.com/2023/the-fediverse-llms-and-rate-limiting/","summary":"Both Reddit and Twitter made policy changes over the last couple of months to restrict API access purportedly due to large increases in LLM training traffic.\nCentralized Social Media I was just refreshing myself on the Reddit situation timeline on Wikipedia . Reuters reported February 14 that Reddit was looking to IPO in the second half of 2023. April 18th brought the announcement that Reddit would be charging for API access after a 60-day notice period ( Reddit , The Verge ).","title":"The Fediverse, LLMs, and Rate Limiting"},{"content":"I went out on a limb and signed up for a book club , hoping to glean some useful concepts and see what interesting people I might meet at discussion time. As a side note, this was the first time I had heard about Chatham House Rule . It lays a groundwork for an open discussion of ideas and kind of reminds me of open-source software licenses. 😃\nWhen a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be revealed.\nThe committed participants voted on the book, and Team Topologies was selected. If I had to pick three main takeaways from the book, they would be the following:\nThe far-reaching implications of Conway\u0026rsquo;s law A focus on balancing team cognitive load when working through team responsibility boundaries The realization that a high levels of collaboration is not always a good thing Below I go deeper into these ideas and share some concepts that were insightful and helpful to me.\nChapter 1 There are other structures besides the org chart that influence how work gets accomplished day-to-day. Influence, trust, and friendships drive communication patterns, and communication patterns drive software architecture. Conway\u0026rsquo;s law is a core idea in this book. It states that an organization\u0026rsquo;s software will look like the communication structures of the organization. This takes into account both the formal (and sometimes to a greater degree) informal communication structures.\nWe\u0026rsquo;ll circle back on cognitive load in later chapters, but there is a warning in this one to be careful and explicit about planning for it. Some teams are victims of their own success. They deliver high-quality work and keep operational toil (from the Google SRE book) low. Because of their proven track record and ability to take on more work, their scope starts to increase. If that continues and the cognitive load of everything under the team\u0026rsquo;s responsibility gets too broad, they will start to have specialists on the team. This can lead to bottlenecks and an overall reduction in throughput. Chapter 8 talks about the feedback cycle organizations need and when it makes sense to split a team to maintain the desired scope of responsibility.\nChapter 2 Conway\u0026rsquo;s law is important enough and fundamental enough to effective software design that there have been extensions and corollaries examining various of its implications. One quote by Ruth Malan says:\nIf the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins.\nThis implies that, to begin with the end in mind, software architects should be the influencing team organization as it is fundamentally an architecture question. Chapter 6 lists several common \u0026ldquo;fracture planes\u0026rdquo; where organizations could choose to place their software and team boundaries. Consciously aligning communication patterns at the start of a large initiative is referred to as a reverse Conway maneuver. It attempts to get out ahead of the software being written and shape the interfaces by team organization. This can be easier than trying to fight the natural tendencies later. Low or no communication between certain teams can be a sign that the architecture is working well. If two teams that shouldn\u0026rsquo;t need to coordinate are communicating, that can be a sign that an interface is not quite right.\nChapter 3 This chapter started out with a quick section on Dunbar\u0026rsquo;s number and some generic benchmarks:\n5 people: close personal relationships 15 people: deep trust 50 people: mutual trust 150 people: remember their capabilities On the low end, these relate to Amazon\u0026rsquo;s popularized \u0026ldquo;two-pizza teams.\u0026rdquo; As the numbers increase, the concept moves to how teams can interact and share information together. A high scale example is Amazon\u0026rsquo;s principal engineering community . In 2014, they were at 10k engineers and 300 principal engineers. This let the group still all get together for off-sites and somewhat know other people. The post author estimates in 2021 that has grown to 70k engineers and 2-2.5k principals. That requires a different sub-grouping strategy to have any type of relationship.\nKeeping Dunbar\u0026rsquo;s number and human relationships in mind, that trust should lead to a team-first mindset. It is the team that owns the software and the outcomes. Putting the team first (and not competing within the team) will result in the best long-term outcomes. The team should rise and fall together. Figuring a way to reward at the team level encourages team accomplishments and not people trying to be heroes.\nThere are several types of cognitive load, and some are healthier than others. Kind of like trans fat is worse than saturated fat, which is worse than some types of unsaturated fat.\nExtraneous cognitive load (useless peculiarities) has no value and should be eliminated as much as possible Some level of Intrinsic cognitive load is necessary like knowing your programming language. Minimize when possible through tech stack consistency, pairing, etc. Leave as much budget as possible for Germane cognitive load. This is the domain-specific business problem and is where the most business value is gained. Chapter 4 Try to make teams as static as possible. Shifting people around for initial implementation and leaving others behind for support might not provide an adequate level of ongoing work on that system. Optimize team structure for long-term evolution, not short-term \u0026ldquo;special\u0026rdquo; projects.\n\u0026ldquo;Flow of change\u0026rdquo; optimized, low friction organizations are inversely related to the number of handoffs.\nCorollary: Any governance and security should happen as part of the flow (ideally automated async) and not require a handoff to a person. Platform teams should interact via code interfaces and not through manually filled tickets. Allow devs to create pipelines from a catalog. Permission requests for those apps are service requests (ideally PRs linked to SNOW for auditing). Product teams need support via simplification of other concerns through interfaces (maintained by platform teams) to keep the cognitive load where it should be and focus on domain concerns. Review team dependencies and explicitly accept them or change processes to remove them. DevOps is an organizational mindset and communication pattern more than it is tooling. Have to get the feedback of running system performance back to devs as directly as possible. \u0026ldquo;DevOps teams\u0026rdquo; should be only an enabling (educating) team for a transition period and have an end date. SRE teams are sometimes needed, but only for large scale. Stream-aligned teams can own those responsibilities on a small scale. SRE teams are paired with one or a couple of specific steam-aligned teams. This pairing requires a high level of engineering discipline on everyone\u0026rsquo;s part to not devolve into \u0026ldquo;us vs them.\u0026rdquo;\nQuestion: Just say you\u0026rsquo;re coming from a Dev team plus Ops team. You align the same headcount into two stream-aligned teams. How do you train everyone up to not just have two functions split within the same team?\nChapter 5 The book now gets into the four fundamental team topologies, hence the title.\nStream-aligned teams serve a specific product or customer persona. They are multi-disciplinary for end-to-end software requirements, design, delivery, operation, and security. These have commonly been referred to as platform teams, but the book prefers the term stream-aligned team since it is more broadly applicable. This is because the customer value stream might flow through multiple products, or one product could be broken up into multiple independent work streams. Platform teams abstract things to reduce cognitive load. They should measure themselves on ease of use and value derived from their services. \u0026ldquo;Thinnest viable platform\u0026rdquo; is a mantra these teams should strive for. Enabling teams allow stream-aligned teams to acquire missing capabilities or figure out problems that take more research or trials. They act as educators with the goal of making the stream-aligned teams more self-sufficient. These teams are generally time-bound, at least with respect to each specific team they are serving. They don\u0026rsquo;t do the work, but point out strategies that can be applied. This makes me think of hearing about the AWS Cryptography Bar Raiser program in a Re:Invent talk (not to be confused with an Interview Bar Raiser ). Complicated subsystem teams are formed due to the complexity of algorithms involved in their software, like video codecs. The decision to create this type of team is driven by cognitive load, not the desire to create an interface to share software with multiple consumers. Many companies won\u0026rsquo;t need these teams unless they are doing something very specialized. If platforms are large enough, they might have their own stream-aligned, complicated subsystem, and platform teams within. The platform still appears as a single entity and is consumed via a single API by the top-level stream-aligned teams.\nChapter 6 Speed of delivery necessitates correct team boundaries to enable autonomy. The first half of the chapter covers types of hidden monoliths. Most people are familiar with these. One of the common ones I\u0026rsquo;ve seen a good bit of is the joined-at-the-database monolith. Data dependencies between apps sharing a database make it hard to break apps off and move the system of record elsewhere. Ongoing backfilling of the original database from the new system is necessary to keep the other apps working. That is just for a one-way dependency; it gets almost impossibly complex with multi-master datasets where more than one application modifies shared data.\nFracture planes are lines where you can cleanly split software when dividing responsibilities between stream-aligned teams.\nBusiness domain: As popularized by the book Domain-Driven Design , this approach attempts to put different areas of the business into bounded contexts. Examples could include billing, sales, underwriting, etc. This is one that is unlikely to be done exactly right on the first shot, so it will need to morph over time as better boundaries are found. Regulatory or risk level: An example of this would be PCI-DSS systems in a retailer. Performance isolation: At work, we keep performance-sensitive things separated from other processes that have much lower SLAs. We don\u0026rsquo;t want a process that needs to run within two days to impact a process that needs sub-second response times. Time zones for geographically split teams: This fracture plane harkens back to Conway\u0026rsquo;s law and the desire to maintain efficient communication within a team. I remember the AWS Firecracker microVM team being all located somewhere in Europe. I think it was this office in Romania . Others: change cadence, technology, user personas, etc. Chapter 7 Semi-orthogonal to the four team topologies, there are three interaction modes for teams: Facilitating, Collaborating, and X-as-a-service. We\u0026rsquo;ll get facilitating out of the way first. This is most often correlated with Enabling teams helping educate other teams.\nThe other two are a spectrum of how much \u0026ldquo;entanglement\u0026rdquo; teams have with each other. For high flow, X-as-a-service should be the primary interaction mode. It is defined by good APIs between teams where one team can use another\u0026rsquo;s product without human coordination between the teams. Services flow \u0026ldquo;up\u0026rdquo; the abstraction stack. Platform teams provide services to complicated subsystem and stream-aligned teams, and complicated subsystem teams provide software components to stream-aligned teams. X-as-a-service interaction patterns scale well across multiple consumers, which is great for platform teams at large companies. They could have tens or hundreds of stream-aligned teams using their platform.\nFor times when good APIs don\u0026rsquo;t exist between teams, Collaboration mode is needed. This allows for rapid exploring of the problem space, building structures and abstractions, and then assigning ownership of the gray area to one team or the other. Much progress can be made during this time when teams are \u0026ldquo;pairing\u0026rdquo; with each other, but it comes at the cost of a higher cognitive load on both teams. Each team needs to partially learn the space of the other. One requirement of success is both teams need joint responsibility for the result, which leads to everyone pulling in the same direction and higher trust. Unlike X-as-a-service, a team should not be in collaboration mode with more than one team at a time. This would spread them too thin and force them to take on more cognitive load than is manageable.\nOne fascinating bit of research was from the referenced paper How Intermittent Breaks in Interaction Improve Collective Intelligence (Bernstein et al.) It finds that while collaboration increases the average solution quality, the maximum quality is only reached with intermittent collaboration. Those involved can benefit from the group\u0026rsquo;s shared understanding but still have the freedom and be forced to think on their own. This leads to less groupthink, more divergent exploration, and a larger possible solution space covered. The concept reminds me of an experimental design class in college taught by Professor Long and hyperparameter tuning in ML models. The right level of randomness and exploration of the solution space will lead to an overall better result than getting stuck on a local maximum.\nChapter 8 Sensing is critical for adjustments as needs change. Teams should be frequently analyzing their dependencies on other teams and explicitly decide if that is what makes sense for their goals or if change is needed. A boundary with one specific other team might be getting more well-defined as a result of collaboration. It could be time to pull back from that collaboration relationship and use that single collaboration quota in another area. Company and software footprint growth is a common driver of team splits to maintain the right level of cognitive load.\nAnother critical aspect of sensing is ensuring that operational insights and pain points inform development cycles. Not having that feedback cycle hurts productivity in the long run due to maintenance overload. A mechanism for adequately sensing in this area is error budgets as covered in the Google SRE book. Similarly, keeping new development separate from legacy system teams reduces the ability of the legacy systems to adopt newer telemetry and other techniques.\nApplication The concepts in this book closely aligned with some situations I have experienced in the last couple of years. As our cloud environment grew, in 2020 I moved off the Architecture team to start a new Cloud Engineering team. Then about a year later we started a Data Modernization program with a big focus on moving our data into Snowflake with modern data pipelines and analytics tooling. A high level of collaboration was needed between Cloud Engineering and the newly formed Data Engineering team as we built out that environment from the ground up. We utilized that collaboration to accomplish a lot. I used my existing relationship with the Security team to be the project liaison for security design reviews. We built out Snowflake and Talend infrastructure platform, data lake zone definitions, an ELT framework with CI/CD, serverless ML Pipelines , Eventbridge custom event integrations, to name a few.\nAfter working closely with that project for almost two years, an opportunity came up to move back closer to software development on the Integrations team. After discussing with my manager, I applied for and accepted that position with the plan of staying for another three months while we hired some more Cloud Engineers. During the knowledge transfer process, I could see that we had probably gotten a bit farther than ideal into process entanglement between Cloud and Data Engineering teams. I had effectively become a Data Engineering team member with Cloud Engineering background and permissions. There was some decoupling we had to do over those months like me being less involved in design sessions, stopping going to Data standups, etc. In the end, we had a relatively smooth transition. There is still some more work to be done balancing separation of duties with not overloading the Cloud team with injects, but we are making process progress.\nThis process and reading the Team Topologies book did lead me to realize the care that needs to be taken when skirting not \u0026ldquo;staying in your lane.\u0026rdquo; The three-month transition worked well, but it would have been rockier on a short transition. You don\u0026rsquo;t want to leave a \u0026ldquo;you-sized hole\u0026rdquo; that makes it hard for you to move on to another opportunity or leave your team in the lurch since it is hard for them to replace your specific shape of responsibilities.\n","permalink":"https://blog.danieladamstech.com/2023/team-topologies/","summary":"I went out on a limb and signed up for a book club , hoping to glean some useful concepts and see what interesting people I might meet at discussion time. As a side note, this was the first time I had heard about Chatham House Rule . It lays a groundwork for an open discussion of ideas and kind of reminds me of open-source software licenses. 😃\nWhen a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be revealed.","title":"Team Topologies"},{"content":"When I started writing, I anticipated it would be mostly text with the occasional diagram included. But my most recent post was long enough I figured a picture might make it a bit more interesting.\nI knew I could store the images in my Hugo static site repo, but I didn\u0026rsquo;t want the over-time slowdown caused by having to clone even small images. I wanted to host them outside the posts repo. There\u0026rsquo;s a chance in the future I will create a separate repo using LFS for content tracking and a CI job for uploading them, but manual uploads will suffice for now.\nI started by looking at Cloudflare Images. I had read some blog posts about that service, and it sounded like a nice API. Upload you images, and the service automatically resizes them into variants that can be requested by alias for different resolutions. The pricing had a floor of $5 a month instead of strictly per-request cost. I\u0026rsquo;m sure it is a great, developer-friendly service, but I am only planning on every 4th post having images or around 12 images uploaded a year. At that point, I wanted something that would be basically free. I ended up going with Cloudflare R2 storage (like S3), configuring https://images.danieladamstech.com to serve the bucket, and pre-computing the variants client side.\nCloudflare R2 is a storage service with zero egress fees and a S3 compatible API. This docs page helpfully shows which portions of the API are implemented. The biggest feature missing in my mind is object versioning. I love S3\u0026rsquo;s versioning in combination with a lifecycle to expire noncurrent versions after 30 days. This provides a convenient \u0026ldquo;recycle bin\u0026rdquo; to help prevent accidental deletion.\nMany of my iPhone camera pictures are in heic format which needs conversion to jpg for display in browsers. I found this article about how to use the Mac built-in tool sips for image conversion. Then I read that imagemagick had better results and is open source so ended up using that.\n1 2 3 4 5 6 7 8 9 10 #!/bin/bash set -eu SOURCE=\u0026#34;$1\u0026#34; TARGET=\u0026#34;$2\u0026#34; # blog text width magick \u0026#34;$SOURCE\u0026#34; -format jpg -resize 720 \u0026#34;$TARGET-small.jpg\u0026#34; # 4k width magick \u0026#34;$SOURCE\u0026#34; -format jpg -resize 3840 \u0026#34;$TARGET-large.jpg\u0026#34; ","permalink":"https://blog.danieladamstech.com/2023/blog-images/","summary":"When I started writing, I anticipated it would be mostly text with the occasional diagram included. But my most recent post was long enough I figured a picture might make it a bit more interesting.\nI knew I could store the images in my Hugo static site repo, but I didn\u0026rsquo;t want the over-time slowdown caused by having to clone even small images. I wanted to host them outside the posts repo.","title":"Blog Images"},{"content":"We interrupt our regularly scheduled programming to bring you this breaking news\u0026hellip; Truly. I was going to write a post on Team Topologies I read for an upcoming book club . Now, that will be pushed to next week. :)\nPreconditions Our hot water heater had stopped working, and we were thinking it was probably the heating element. I got some advice to at least call a plumber and see what they would charge since it was a bit tedious to do the repair. When the plumber came out, he also checked the house\u0026rsquo;s water pressure. It was around 95 PSI when it should have been under 80 to prevent damage to the interior plumbing. We knew we wanted the hot water heater fixed, and the water pressure issue seemed to be a problem waiting to happen if not addressed.\nROI Analysis The quote came back later that day at $984 for hot water heater repair, expansion tank replacement, and pressure reducing valve replacement. Add on another $225 for a second expansion tank replacement in the crawlspace, and that puts the total to $1,200.\nMy initial estimate for the DIY route was $230 in parts and $110 in tools for a savings of around $850. That included a splurge on a not-completely-necessary Klein clamp multimeter . I wanted to verify the hot water heater was turned off without touching it with voltage probes since it runs on 240 volts and I wanted to be super careful. I could have bought a $10 clamp ammeter at Harbor Freight, but I took the opportunity to upgrade my Amazon multimeter to a nice Klein that can also measure DC amperage with the clamp. That might prove useful for some future ham radio projects.\nRisk Analysis I was well aware of how damaging water can be to a house, so part of the condition for taking on this plumbing project was ensuring limited damage potential if a small leak happened in the garage or crawlspace. I hedged my bets by building a gallon drain bucket with a hose attachment at the bottom like a rain barrel. I didn\u0026rsquo;t know how much would be left in the pipes after running the faucets and hose spigots dry. Theoretically, this would allow me to drain an unlimited amount of water from the pipes. Little did I know the magnitude of what would transpire!\nOnce I felt prepared on the electrical safety and water drainage fronts, I was ready to go.\nImplementation I split the project into three phases: hot water heater, pressure reducing valve, and expansion tanks. I tried to do them when my wife and kids were out of the house since the tasks all involved turning off the whole house water supply.\nThe hot water heating element replacement went smoothly. I had done my research ahead of time and bought a purpose-built heating element wrench . The elements were torqued in to prevent leaks, so the wrench was necessary to get leverage in the tight space. A monkey wrench or channel lock pliers would not have been successful. I got a decent bit of sediment out of the tank when I drained it. I read that if you drain the tank regularly, removing sediment is good. However, if the tank has not been drained in a long time, removing sediment can actually weaken the walls. Since this hot water heater was just 4 years old, hopefully we\u0026rsquo;re on the good side. I replaced the elements with low watt density models that folded back for more surface area. Dispersing the same wattage of heat over more surface area should give the elements longer service life.\nNext up was the pressure reducing valve. This project had a couple of starts and stops. I originally replaced the valve using the fittings from the old one. Nice and simple! However, when I turned the water back on, I saw a drip every 2-3 minutes. This had been happening on the old valve and I think was the root cause for the adjustment bolt rusting over. I went to Lowest to get a PEX crimp tool , pipe, and a straight coupler. After verifying the ring didn\u0026rsquo;t just need another crimp, I turned the house water off again and disassembled everything. I tried to stage it all up with the new pipe section and new couplers, but the strain was just too much. To avoid yet another trip to the hardware store, I got out my heat gun to see if that would help me bend the PVC . It didn\u0026rsquo;t work as well as that guy, so I just went back out to get a 90 degree coupler. After installing that, I had perfect strain relief. Another perk was that 90 degree elbow ended up being the low point of the pipe on that side. If there was a drip in the future on that side, it would run with gravity away from the valve instead of over it.\nLast phase to go - expansion tanks! These were lower priority than the hot water heater (ASAP) and pressure reducing valve (as soon as convenient.) I had already filled the expansion tanks to 60 PSI to match the pressure after the pressure reducing valve. Laura and the kids were gone one night at Vacation Bible School at her parents\u0026rsquo; house, so I used that opportunity to finish my project. I went to taekwondo after work from 5:00-7:00, was home by 7:30, and had eaten dinner by 8:00. The hot water heater expansion tank was a breeze. I just had to shut off the cold side valve of the hot water heater, drain the hot from the faucets, and drain 30 seconds out the hose attachment at the bottom. That tank was still good since it was holding air pressure, but I figured since I had bought the part and taken the old one off, I would stick with the plan. I told myself once I was done with this project I\u0026rsquo;d have a basically refreshed hot water and pressure regulation system.\nAt this point in time, it was 8:45 and I was pretty tired. I thought about calling it a night, but I really didn\u0026rsquo;t want to drag the project on when I had the chance to finish it with just one more replacement. The hot water heater expansion tank was so easy that I just had to finish the other one out\u0026hellip; So I went for it. I turned off the whole house water shutoff from inside just like before when I was working with the pressure reducing valve. This one was a bit of a struggle to get taken off. There was a metal strap wound around the tank to support it, and the ~22lb weight of the filled 2 gallon tank on its side was not spinning freely. I would unscrew it about an eighth of a turn, lift it up to allow the support to uncoil, and then reposition the wrenches for another turn. This was relatively tiring due to the positioning in the crawlspace and the weight of the tank. Finally I get it off, and water starts spurting out of the pipe! This was not what I had expected. Everything inside the house and both outdoor spigots had zero pressure. I did not anticipate it, but evidently this was upstream of the indoor whole house shutoff. That meant it was above the pressure reducing valve, so I had a 3/4 pipe flowing at over 100 PSI. Not good. It was filling up my 5 gallon bucket much faster than the non-pressurized hose exit port was draining. I had another mop bucket that I was crab-running back and forth to the crawlspace exit while I was trying to think about what to do next. After a couple minutes of doing the same thing and seeing the same results (that not-Einstein quote), I figured it must be a live pipe and not any type of drainage. I grabbed a wrench and ran up to the road to shut off the water main. Thankfully that worked! I went back under the crawlspace and sat in the water for a minute to recover. Thankfully the water main valve was good and fully shut off the water so I could get the new expansion tank put on. I couldn\u0026rsquo;t get the steel pipe hanger to loop over on itself for a friction fit quite the same way as it had before, so I tied a wire around it and the pipe to keep it positioned in the correct spot. Turning the water back on and verifying no leaks was completed by 10:30. At this point, the crawlspace had a large amount of water in it. If I caught half of the water in the buckets, there was probably 25 gallons on the floor. Some had run between the plastic sheets into the ground, but the majority was sitting on the plastic sheets. I spent the next two hours using beach towels to clean up that mess. It turns out the expansion tank under the house had gone bad and the bladder completely filled with water. That was a small consolation that the mess I caused was somehow worth it.\nReflections Clean up your mess! Don\u0026rsquo;t let someone else (or your future self) bear the consequences of your learning and experimentation. I spent two hours sopping up standing water on the plastic sheets in the crawlspace to avoid mold. In software engineering, be sure to follow up and clean up any technical debt or adverse downstream effects that your experiment caused.\nI had some ah-ha moments during my research for this process. I have heard the voltage is water pressure and amps are volume of flow analogy before. One youtube video talked about water flow vs pressure and how reducing water flow by halfway shutting a cutoff valve does not actually solve the pressure problem. There has to be a pressure reducing valve to drop pressure from the street level to the necessary house level. It was neat to hear the applicability of those physics concepts in their native domain besides just an analogy for something else.\nI spent roughly 15 hours on this project: 4 for hot water heater elements replacement, 7 for the pressure reducing valve replacement, and 4 for replacing the expansion tanks. I could have hurried more but tried to take it at a measured pace and not rush. I ended up spending about $500 including the additional PEX piping, adapters, and crimp tool. This resulted in a final savings of $700 our an hourly rate of $45. It took longer than expected, but the cost savings was only one goal. The other desire was learning more about how things work and setting myself up with new skills for the future. The project definitely fulfilled the learning objective, so I\u0026rsquo;m calling it a win.\n","permalink":"https://blog.danieladamstech.com/2023/in-pursuit-of-learning/","summary":"We interrupt our regularly scheduled programming to bring you this breaking news\u0026hellip; Truly. I was going to write a post on Team Topologies I read for an upcoming book club . Now, that will be pushed to next week. :)\nPreconditions Our hot water heater had stopped working, and we were thinking it was probably the heating element. I got some advice to at least call a plumber and see what they would charge since it was a bit tedious to do the repair.","title":"In Pursuit of Learning"},{"content":"Today we will follow up on my previous post about programmable platforms and delve into WebAssembly as an implementation option. I have read about WebAssembly on the CloudFlare blog since 2019, but my interest piqued the other day when listening to the Software Snack Bites podcast about WebAssembly . The 1.0 spec was published in December 2019 which started a marked uptick in awareness. Notable implementations include Wasmtime by the Bytecode Alliance (writers of the spec) and the CNCF project WasmEdge .\nFrom the spec: WebAssembly (abbreviated Wasm) is a safe, portable, low-level code format designed for efficient execution and compact representation. Its main goal is to enable high-performance applications on the Web, but it does not make any Web-specific assumptions or provide Web-specific features, so it can be employed in other environments as well.\nWebAssembly was originally developed for browser execution of optimized C++ via JavaScript bindings. As the specification solidified, other people saw promise on the server side. WebAssembly has several nice properties for serverless multi-tenancy. It is secure by default and only exposes interfaces through capabilities. On the performance side, it has \u0026ldquo;zero cold start penalty.\u0026rdquo; This is HUGE for serverless platforms. One of the tradeoffs a developer must account for when deciding whether to go serverless or not has traditionally been acceptable p99 latency due to cold starts.\nWebAssembly was designed for web levels of forward compatibility. If there were any backwards incompatible changes, a new binary format version would be created, but the expectation of that is \u0026ldquo;very infrequently, if ever.\u0026rdquo; This is good news for both application and tooling developers, who can expect a stable set of system interface specifications that will not change.\nBuilding platforms is a natural fit for Wasm. SaaS customers need ways of embedding custom logic in workflows within the platform. One way of delivering that capability is through rules engines or custom DSLs , but those are more constrained in the capabilities they offer. Securely and performantly executing customer-provided code provides the open-ended avenue necessary to unleash developer creativity.\nCorrectness of custom code is always the responsibility of the developer, but platforms hosting user code can take away much of the other complexity. The user does not have to worry about reachability and latency of API calls from the platform out to his endpoint. The \u0026ldquo;glue\u0026rdquo; is all managed by the platform host and generally provides the best end-user experience. A good example of platform-hosted and external extensibility in the same platform is Snowflake\u0026rsquo;s Snowpark UDFs and External Functions . Sometimes a requirement for exclusive control might keep you from putting a piece of functionality into a platform, but in most other cases, the reliability and performance lean strongly towards submitting user code for the platform to manage and run.\nI would like to call out two examples of these in the real world. Shopify functions allow developers to customize the behavior of the platform ranging from discounts , payment customizations , custom validations , and more. Targeting WASM allows developers to pick the most appropriate language for the task. High level logic can quickly be written in JavaScript or TypeScript, and computationally intensive operations can be optimally implemented in Rust. Picking 10 of the highest customer-requested \u0026ldquo;logic choke points\u0026rdquo; within the app and opening those up for extensibility can remove large burdens from the SaaS development teams. Customers that were previously blocked and have the resources to invest in customization can self-serve.\nThe other instance of Wasm support is in a \u0026ldquo;platform for platforms.\u0026rdquo; It\u0026rsquo;s like platform-ception! Cloudflare Workers for Platforms provides a managed execution environment for customer-supplied code with developer-friendly zero-cost abstractions . They allow Workers for Platforms customers to have an unlimited number of scripts so all the end users can submit their customizations. If I was building a SaaS application, I would be very interested in trying this out.\nAs always, there is no completely free lunch. Platform hooks are just like any other API in that they must be kept stable. Increasing API surface before you know you have the correct abstraction can lock you into a design and prevent refactoring down the road. This interview with Nathan Sobo about building the Atom editor at GitHub (and his new editor Zed ) highlights this. Looking back, Nathan said they focused on extensibility a little too much when building Atom. Some of those decisions, like allowing extension code to run on the main thread instead of VSCode\u0026rsquo;s more constrained Language Server Protocol design, lead to inflexibility down the road.\n","permalink":"https://blog.danieladamstech.com/2023/wasm-for-platforms/","summary":"Today we will follow up on my previous post about programmable platforms and delve into WebAssembly as an implementation option. I have read about WebAssembly on the CloudFlare blog since 2019, but my interest piqued the other day when listening to the Software Snack Bites podcast about WebAssembly . The 1.0 spec was published in December 2019 which started a marked uptick in awareness. Notable implementations include Wasmtime by the Bytecode Alliance (writers of the spec) and the CNCF project WasmEdge .","title":"Wasm for Platforms"},{"content":"Back in April, I read one of Gergley\u0026rsquo;s newsletters on Steve Yegge and Developer Productivity . It was a very insightful and enjoyable read, but the part that stood out to me was not the main topic of the post. It was a semi-famous (but unknown to me) piece Steve wrote after six years of tenure at both Amazon and Google. Known as Stevey\u0026rsquo;s Google Platforms Rant , it contrasts Amazon\u0026rsquo;s and Google\u0026rsquo;s execution and mindset. Even though Google did almost everything in a technically superior way, Amazon came away with the winning platform.\nAmazon stressed that their internal APIs between their internal product teams must be the same level of quality as external product APIs. Steve\u0026rsquo;s piece lists challenges and learnings along the way, but the end result was a programmable platform. The focus on solid internal APIs mirrors Chick-fil-a\u0026rsquo;s Enterprise Architecture principle Design for Composability .\n\u0026ldquo;A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.\u0026rdquo; - Steve Yegge\nOne of the biggest reasons to build a platform is you cannot please everybody. Users\u0026rsquo; ability to extend and tweak your software could be the difference between it fitting their use case or not. Two types of platforms exist. Some are services other developers will build apps on top of, like S3. These were never meant to be used in isolation. Another variety is an application designed to be extensible by the end user, like Shopify. The user can begin with the out-of-the-box experience and later customize it with hooks.\nI read Steve\u0026rsquo;s Platforms post after adding event-based triggering to my ML Pipeline. In 2022, I built a ML Pipeline to provide orchestration and MLOps for several models the Data Science team was ready to deploy to production. After an initial MVP, we wanted to move from static schedules to triggering on Snowflake table load events. There were a variety of conditions when we wanted to run training, inference, or both. For example, one time series model needed retraining monthly or when specific external datasets were updated. We could have written a rules engine and tried to describe all the conditions as rules. However, we saw the probable trajectory of increasingly complex conditions on the horizon. After thinking about it for a couple weeks and gathering input from various teams, I decided to implement the event filtering with a Data Science controlled code hook. This enables programatically deciding which phases to run on any given event. Reading Steve\u0026rsquo;s thoughts on platforms a couple weeks later gave the satisfying feeling we had picked the right direction.\nThe orchestration code hook is stored in the ML model git repo and managed by the Data Science team. A custom Cloudwatch event of type data_lake:table_loaded triggers one of the ML Pipelines, and the Step Function will invoke the code hook to determine which phases to run. These phases include data prep, training, inference, and post-processing. The hook is executed in a Lambda function, and its interface is modeled after the Lambda handler. The event passed to the code hook is the unwrapped Eventbridge event. The context parameter contains properties like ML model name to allow code sharing across models without modification. Future iterations of this interface will add other methods to simplify consumer code like get_snowflake_connection(). After preparing the event and context objects, the Data Science team code hook file is imported via a dynamic module load as specified in the python importlib docs . The result returned from the function call determines choice states later in the Step Function to select specific phases to run. We have started using this system for basic conditions, but the sky is the limit for the future. The determination can be based on data drift, upstream data quality, or any other future requirements.\nNext week - how WASM enables Platforms !\n","permalink":"https://blog.danieladamstech.com/2023/platforms-not-products/","summary":"Back in April, I read one of Gergley\u0026rsquo;s newsletters on Steve Yegge and Developer Productivity . It was a very insightful and enjoyable read, but the part that stood out to me was not the main topic of the post. It was a semi-famous (but unknown to me) piece Steve wrote after six years of tenure at both Amazon and Google. Known as Stevey\u0026rsquo;s Google Platforms Rant , it contrasts Amazon\u0026rsquo;s and Google\u0026rsquo;s execution and mindset.","title":"Platforms, Not Products"},{"content":"Two things came together to lead me to start this blog. Exploring new tech has been a desire for a while. The other, writing, is a recent realization of necessity. My personal goal is to write one post a week for the next year.\nWriting The more contemporary driving factor is improving my writing. Several sources I\u0026rsquo;ve read and listened to lately have emphasized the importance of writing. Brian Chambers, chief enterprise architect at Chick-fil-a, has advocated for writing in public . One of the resources linked in his post is Matt Rickard\u0026rsquo;s blog , which I now follow. Brian\u0026rsquo;s description is:\nMatt writes every day and is incredible at cross-disciplinary pattern matching. I was amazed to learn he writes these super-technical posts in ~20 minutes a day. 🤯\nMatt\u0026rsquo;s daily writing of synthesized thought is amazing! \u0026ldquo;Cross-disciplinary pattern matching\u0026rdquo; reminds me of the book Range my brother got me for Christmas several years ago. I haven\u0026rsquo;t read enough to possess that skill set, but it is a long-term goal. The Staff Engineer\u0026rsquo;s Path (recommended by Gergely Orosz ) champions writing as the way to scale influence across a large tech org. Several recent Changelog episodes featured writers who encouraged others to develop that skill as well.\nAnd we all know about Hamilton\u0026rsquo;s writing thanks to Lin-Manuel Miranda\u0026rsquo;s compositions!\nTech The other half of the equation is playing with new tech. I have used AWS Lambda for a long time, but have been itching to try out some other serverless application building blocks. I have been following Cloudflare\u0026rsquo;s blog for 4 years for their excellent tech articles. By the way, they write the most comprehensive transparent postmortems, a fact that Gergely highlighted when analyzing the DataDog outage .\nI created this blog using a static site hosted on Cloudflare Pages . I had looked at static site generators several years ago, but never ended up creating a website at that time. Hugo was popular then, but now 11ty seems to be the top one for templating flexibility. I still stuck with Hugo for its single-binary simplicity.\nThis site and the Terraform for Cloudflare configuration are public on my GitHub at blog-danieladamstech and terraform-danieladamstech-cloudflare . I\u0026rsquo;d like to keep the Terraform public as long as possible. For now, it is relatively non-sensitive, but I might create new app configs in a private repo as needed.\nIn the future, I want to try building an API with Cloudflare Workers . No cold starts would remove a major traditional downside of adopting serverless. Cloudflare is also innovative in the application zero trust architecture space. I would like to host an app on my Raspberry Pi and use Cloudflare Tunnels with Access to expose it without opening up my home network to direct internet ingress.\nThanks for reading, and hope you enjoy future posts!\n","permalink":"https://blog.danieladamstech.com/2023/hello-world/","summary":"Two things came together to lead me to start this blog. Exploring new tech has been a desire for a while. The other, writing, is a recent realization of necessity. My personal goal is to write one post a week for the next year.\nWriting The more contemporary driving factor is improving my writing. Several sources I\u0026rsquo;ve read and listened to lately have emphasized the importance of writing. Brian Chambers, chief enterprise architect at Chick-fil-a, has advocated for writing in public .","title":"Hello World"}]