Tag Archives: microservices

REA Tech Blog Scribbles

Posted in Uncategorized by

About

Just going through REA’s blog in no particular order as of September 15, 2016.

Getting Shit Done

Well-written: http://rea.tech/getting-shit-done/

What prevents us from “Getting Shit Done” is the last time we got shit done.

Pair programming lessons

  • There is only so much we can learn by watching someone else type
  • Driving helps us gain confidence as a member of the team and feel comfortable working with our pair
  • It gives us a sense of ownership
  • The fastest way to feel comfortable making mistakes, is to do it while someone is watching
  • Breaks comfort zones (less slacking, pushed to learn more)
  • Take breaks
  • Be prepared to say “I don’t know”

Why team managed infrastructure makes sense

  • Conway’s Law states that as an engineer you inevitably create a system that has the same structure as that of your organization.
  • Increases autonomy, responsibility, ownership, satisfaction and skills
  • Developers become dev-ops and support 24/7, incentivized not to make mistakes
  • Anyone in team should be able to deploy to production at any moment
  • Teams free to choose their own solutions, but should communicate/share with other teams
  • Some holistic solutions/standards will need to be adhered to by teams as deemed necessary

Category Theory

http://rea.tech/how-we-used-category-theory-to-solve-a-problem-in-java/

The Six QA Hats

tmp

Micro services, what even are they?

  • Different intent that SOA. SOA describes the organisation business logic, micro services describe the services themselves
  • Should do one thing well (narrow responsibility), have a small codebase (rewrite and redeploy within 2 weeks) and can be written in any language (cross-cutting logging and monitoring)
  • Mitigates bad design by restricting to a small service
  • More complexity in overall architecture, responsibilities and data integrity (scattered databases!)

Micro services are just tiny, focused SOA services. Too many micros and you push the complexity from the codebase to the architecture (contracts, responsibilities, data integrity). Like most things, use experience and common sense to do what’s right for a project.

A microservices implementation retrospective

What is the right size for a microservice? Varies between projects — balance between complexity (lots of repos, builds, clusters), maintainability (shared functionality, code) and the traditional benefits of SOA.

  • REA template projects
    • Logger (aggregation and rotation configs)
    • HAL (index and health check) and HAL Browser gems for exposing links between resources (i.e. RESTful HATEOAS)
    • Sample environment variable, database connection, async job, deployment config and monitoring
  • Non-blocking, idempotent (i.e. calling repeatedly is not harmful) APIs exposing meaningful events
  • Pact for contract testing
    • REA-developed open-source Ruby gem
    • Enables CDC (Consumer Driven Contract) testing (i.e. providers list set of endpoints, consumers inform provider of what services/data being used). Suited for when you control the provider and consumer.
    • Used for unit tests and mocks only, not ‘real’ integration testing
    • Checks consumer and provider contracts are in sync using a generated-then-shared pact file
  • Data integrity through idempotent job retries, job monitoring, change events triggering retrieval of current data, clear source of truth systems, unidirectional data flow (i.e. one-way sync).
  • Aggregated logging across many modules through Splunk (aggregation) and Nagios (monitoring). New Relic could also have been adopted.

Personally I’m not currently a huge fan of hyperlinked resources (i.e. HAL). They add extra weight to responses which 99% of the time will be ignored as users don’t tend to browse JSON and developers tend to read the documentation. Also, endpoints rarely change so additional calls for dynamic support are inefficient. My preference is RESTish or EBTH (everything but the hyperlinks).

Distributed Agile — Extreme Communication

  • Email is not the primary source of communication (face-to-face, video conf, IM)
  • Always-on video conferencing for offsite teams (China)
  • Remote pairing
  • Virtual card walls, physically replicated

Really interesting that REA needs so much development power that they need to outsource, which is never better except for cost-cutting. Pioneering the new global workplace I suppose!

Feeling Validated – A different way to validate your input

  • Validation requirements constantly change = refactoring
  • Use a Validation abstract class that only has two concrete subclasses: Success and Failure (with message)
  • Chain validation using flatMap. It will only continue to the next validation if passed the previous validation (otherwise map is empty) — and eventually returns a Success object.
validateInput(Map("postCode" -> "3121"), "postCode")
.flatMap(postCode => postCode.parseInt.leftMap(nfe => "post code must be a number"))
.flatMap(postCode => postCodeLookup(postCode).toSuccess("not in a valid area"))

For comprehensive validation, add metadata (i.e. annotations, directives) to elements and auto-validate.

The abject failure of weak typing

  • Ruby play-dough typing
  • Types are a proposition (i.e. assertion) applied by the compiler and known up-front. Implementation is it’s proof.
  • Tags are runtime type information (e.g. string, array, map) used by dynamic languages
  • Dynamic languages are actually unityped, not untyped
  • “Don’t write software that isn’t broken; write software that can’t be broken.”
  • “It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.”
  • Algebraic Data Type (ADT) are an abstract set of ‘objects’ and their operators. Very flexible objects that can return many types and apply functions depending on type. Objects with finite possibilities — a set. Still don’t really understand…!
  • Nulls
    • Ticking time bombs of NullPointerExceptions
    • Employ default values (e.g. “”, 0) or a safe Null object (that can respond in a sensible way)
    • What does “initialised piecemeal” mean?
    • Write immutable objects initialised in the constructor (to avoid null objects)
    • Java 8 Optional class, which checks for null and can handle missing methods
  • Exceptions
    • Don’t throw exceptions unless unavoidable
    • Checked exceptions are widely despised and frequently ignored (unsubstantiated)
    • When a result could be success or failure, use an Either or Validation (Success/Failure) object instead
    • Define your own ADT
  • Primitives
    • Possible astronomical number of values (e.g. integers). How to test all possible values?
    • Strings are vertitile and often used wrongly. Should only be for unstructured free text, not anything that has a constraint or structure (e.g. name, URL, email).
    • If there are a finite number of possibilities, used an ADT
    • Use wrappers to encapsulate the structure and make it impossible to create invalid variables. Can also be used to normalise as well.

References