wasabigeek's blog

Using the XReal AIR 2 Pros for Remote Work (Programming)

Sat, 31 Aug 2024 00:00:00 GMT

I’d been pondering adding a pair of AR glasses to my remote work setup, as an alternative to a portable monitor. After reading positive reviews and physically trying the XREAL Airs in Japan, I picked up a pair when they went on sale — just in time for a two-week trip where I’d be working remotely. The verdict after two weeks? I’ll keep using them for my travel work setup, but I’m not giving up my 27-inch monitor at home.

The Setup

I got the XREAL Air 2 Pros. The key difference from the base model is that this comes with electrochromic lenses, allowing you to change the tint with the press of a button. The lenses got dark enough that I could skip using the included lens cover.

In addition, I also purchased:

the Beam device - I’m not cleared to install the Nebula software on my work laptop. That would have allowed for screen anchoring, which the wisdom of Reddit said was important. So I got the Beam just for anchoring 😑.
prescription lenses from HonsVR (~US$76 with SPH prescription and a blue light filter). I’m not a fan of contact lenses, so this was the only way for me to comfortably use the glasses.

Using the XREALs unplugged

Don’t be fooled by the marketing, these glasses will not pass off as regular sunglasses in spite of the obvious RayBan influences. They are noticeably larger, and sit much further from your face. Face ID wouldn’t work for me, while it does with my regular glasses.

That said, at 75g, they’re lightweight and comfortable for extended wear. The electrochromic lenses, a key feature of the Pro model, allow you to adjust the tint at the press of a button, eliminating the need for the lens cover. In theory, this lets you move around without having to swap back to your regular glasses.

However, the displays only cover half the vertical area, which creates an unavoidable horizontal line with different tints above and below. Plus, when unplugged, the glasses remain at a medium tint level, leaving you with sunglasses indoors—not ideal. As a result, I ended up switching back to my regular glasses about half the time. While useful in specific scenarios, they’re not something I’d wear outside or as an everyday replacement for my usual glasses.

Using the XREALs without the Beam

When plugged directly into a compatible USB-C device like a Macbook or iPhone 15, the glasses act as an external display, mirroring or extending your device’s screen. The image is sharp and text is very readable, but it can be quite hard to read the edges since those remain in your peripheral vision. The screen moves with your head, so if you want to focus on a corner you have to keep your head still and move just your eyes. As such, it’s great for watching video, but for doing work you’ll have to keep what you’re reading in the center of the screen.

I also learnt my head moves a lot when working, especially while I’m typing. Since the screen follows these movements, it’s a bit disorienting, bordering on nauseating. Due to this, I rarely used the XREALs without the Beam, save for the occasional video in bed or during a long bus ride. Admittedly, the latter was a cool use case; with the electrochromic lenses, I could adjust the transparency to appreciate the scenery. However, that persistent horizontal line remained a drawback.

Using the XREALs with the Beam

The Beam allows you to mirror or extend your device’s display without installing additional software, which sounds great and works, but there are tradeoffs 😞. Compared to a direct connection, the image is noticeably blurrier, though still usable. There’s a subtle flickering, especially apparent on dark backgrounds at lower brightness levels.

In most modes, you can adjust the screen size. I primarily used the default “120-inch” setting, which, in practice, resembled a 27-inch monitor about a meter away—quite satisfactory. Larger screen sizes are possible, but due to the limited field of view, the screens get cropped, requiring you to move your head to see the edges.

The resolution is 1080p, which was acceptable for me, considering my home monitor had been set to 1080p for the past two years without issue. There are three display modes, but I mainly used “Body Anchor.” “Smooth Follow” doesn’t provide a clear view of the edges, and “Sideview” mode seems intended for multitasking, which didn’t feel particularly safe or practical.

It appears the Beam can’t mirror and charge simultaneously. In practice, this wasn’t a significant concern, as the Beam’s battery lasts several hours. As long as I remembered to charge it between sessions, it sufficed.

With screen anchoring, I was somewhat to use my Macbook’s physical display as a secondary screen. Whe the wall was a flat colour, I could reduce the dimming on the lenses while keeping the XREAL display readable and allowing me to read the physical display. However, the aforementioned caveats (the horizontal line and tinting) meant I only used it for secondary functions e.g. keeping an eye on Slack or having Spotify open.

Occasionally, I noticed subtle drifting of the screen when anchored. Restarting the Beam didn’t resolve this; I had to re-anchor the screen periodically until the issue subsided. Also, the Beam gets hot pretty quick, and the built-in fans will start whirling. It’s not crazy loud but noticeable.

UPDATE 2025-02: my beam has started making a weird beeping sound, I’ll have to ask XREAL what’s going on here.

Other Notes

Since you don’t need to physically position an external monitor, the setup occupies much less space and is more ergonomic, allowing you to look forward at a floating screen instead of hunching over a desk.

My remote setup is currently:

a 14-inch Macbook Pro
a Roost laptop stand
the Logitech POP mouse
a folding split keyboard
It’s on the left below, in contrast to my wife’s setup:

With the case and Beam, the setup is only slightly lighter than a portable monitor but significantly easier to pack and transport. Plus, you technically get a larger display, as most portable monitors are around 16 inches.

I still removed the glasses for video calls, though they could serve as an interesting conversation starter.

I haven’t extensively tested Nebula. I briefly tried it on my personal laptop; multiple screens seemed cool, but the quality of individual displays appeared similar to the Beam.

Summary

I couldn’t work a full day in them, but think I got enough benefit out of half a day that I could see myself continuing to use this setup when travelling.

The good

Compact and easy to bring around
A relatively big screen (though 120 inch is a bit misleading, in practice it was like using a 27-inch monitor on a desk)

The meh

Still can’t beat an actual desktop monitor in terms of field of view, comfort and clarity
Beam is kinda annoying to manage
Pricier than a portable monitor, especially if you need the Beam and prescription lenses

If you found this useful and are considering a purchasing the XREAL Air 2s, consider supporting my blog by using one of the following affiliate links:

Lazada Singapore

How I'm dealing with Self-Doubt as a Software Developer

Mon, 24 Oct 2022 00:00:00 GMT

As a software developer, I’ve found self-doubt tends to intensify when:

comparing oneself with others e.g. can I really be as good as person X?
getting some negative feedback, especially from people I’m unfamiliar with (I think this is close to imposter syndrome, where I’m worrying about being judged)
looking at something I don’t know much about or struggle to understand

This is an unstructured collection of resources and thought processes I’ve found helpful when that happens.

Remember we don’t see the journey they took to get there

When comparing yourself to others, it’s super easy to be focused on how good they are now, without accounting for time taken to get there (and how much they struggled, just like you might be struggling now).

I paired recently with @vinistock and was amazed by (from my perspective) his deep knowledge of Ruby internals like singleton classes and module hooks. What helped demystify it was asking:

did he have this knowledge when he joined his current team a year ago? Answer: No, in fact, it was a struggle to learn.
what resources did he use to get better, which he graciously shared. This helped me to chart a sort of “path to improvement”.

There are probably geniuses out there who grasp things immediately, but for the majority of us, it’s about working to get better, and hanging in there.

Realising they made some learning tradeoffs

Another one when comparing yourself to others. @dan_abramov (a public figure in the React community) has written about things he doesn’t know, which helped to break a false notion that because I didn’t know “everything”, I was behind.

It can be easy for us to narrowly focus on what our heroes know deeply about, and forget that they still don’t know everything, and likely made learning tradeoffs to get there.

I still like the idea of being able to know everything, but am a lot more zen about it now. I try to remember that it’s likely something that can be learnt, but I don’t have to learn it now.

Getting more feedback

@bethcodes has characterised imposter syndrome as a lack of feedback, and presented the dangers of choosing not to seek it. To get feedback, we need to put ourselves out there (be it “asking stupid questions”, taking projects outside our comfort zone, or asking directly for it). That’s risky - we can look stupid, get laughed at, and the like. But when we get constructive feedback, it leads to one of two things:

learning what you can improve in
learning that you’re not as terrible as you think

My favourite passage from @bethcode’s post:

On and on, my stupidity was laid bare for the world to see and it turned out it didn’t matter. No one cared, and when I took the feedback on board for next time they thought it was awesome.

Even with rational understanding of the above, it takes me time and repetition to get comfortable with putting myself out there (be it “asking stupid questions” or seeking feedback). Psychological safety is an important pre-requisite, and I think it’s both trying to find a good environment, as well as testing the waters to see if whether people truly practice a safe environment.

On a somewhat related note, working with different people helped as well, as it became clearer that the positive feedback wasn’t one-off. Plus, if many different people are saying similar things, it becomes harder for you to refute. For me, this became clearer when I changed jobs - I really enjoyed my previous company actually, but kept wrestling with my standing in the industry. (I wouldn’t take this at face value though, depending on you / your company you might be able to achieve similar results without leaving!)

Reframing as something that can be improved (Growth Mindset)

I feel a little cheesy saying this - reframing the subject of my imposter syndrome as something improvable has had the most impact over the last year or so. A key catalyst for this was reading Carol Dweck’s Mindset.

One particular example from the book that sticks with me - children were given a set of puzzles to see how they dealt with difficulty:

Confronted with the hard puzzles, one ten-year-old boy pulled up his chair, rubbed his hands together, smacked his lips, and cried out, “I love a challenge!” … What’s wrong with them? I wondered. I always thought you coped with failure or you didn’t cope with failure. I never thought anyone loved failure. … Not only weren’t they discouraged by failure, they didn’t even think they were failing. They thought they were learning.

When inadequacy or jealousy creeps in, I try now to think of the subject as something I can get better in, as opposed to being something impossible. I avoid thinking about whether I can achieve the same level of expertise, just reminding myself that I can get better. I literally imagine and try to emulate a kid saying, “Whoa, that’s awesome! Let me try.”

Again, it’s not something that changed overnight, and I’m still working on it.

Visitor Pattern in Ruby: Examples from Rubocop

Tue, 09 Aug 2022 00:00:00 GMT

In this post, we’ll learn about the Visitor design pattern and it’s tradeoffs by looking at a real world example: Rubocop, a Ruby linting and formatting library.

What is the Visitor design pattern?

The visitor design pattern separates the operation to be performed from a complex object structure (e.g. a graph whose nodes have many types).

This makes it easy to introduce new operations (“visitors”) for the same structure, but comes at a cost - changing the object structure requires all visitors to be changed. So this pattern works best when changes to the underlying structure are rare, and we expect many different operations to be introduced.

What is Rubocop?

Rubocop is a linter¹ and formatter, which is a fancy way of saying that it checks the raw source code against a set of rules (e.g. for common errors or conformance to a style guide) and can autocorrect it. Each rule in Rubocop is called a “cop” and follows a visitor-inspired pattern.

Note that Rubocop doesn’t follow the classical visitor implementation² exactly, but separates concerns similarly. We’ll look at how it tweaks the pattern, before describing the tradeoffs with real-life examples.

Object Structure: Abstract Syntax Tree (AST)

Before running any cops, Rubocop first uses the rubocop-ast library to create an Abstract Syntax Tree representing your Ruby code. This makes the responsibility of cops much simpler: they don’t have to worry about syntax correctness, and because code is pre-grouped and categorised in the tree, it’s easier to know when a linting rule should be triggered.

Let’s look at a simple example:

require 'rubocop-ast'

code_string = <<~CODE
  private def sum(a, b)
    a + b
  end

  sum(1, 2)
CODE
# => "private def sum(a, b)\n a + b\nend\n\nsum(1, 2)\n"

processed_source = RuboCop::AST::ProcessedSource.new(code_string, 3.1)
processed_source.ast
# =>
# s(:begin,
#   s(:send, nil, :private,
#     s(:def, :sum,
#       s(:args,
#         s(:arg, :a),
#         s(:arg, :b)),
#       s(:send,
#         s(:lvar, :a), :+,
#         s(:lvar, :b))),
#   s(:send, nil, :sum,
#     s(:int, 1),
#     s(:int, 2)))

The parsed AST breaks down the code into relevant constructs based on Ruby’s syntax. We can visualize the above:

For now, it’s sufficient to know that the object structure is basically a tree - don’t worry too much if you don’t understand exactly how it works. (For a good overview of how this AST fits into the Ruby interpretation process, check out this great video by @OngMaple. You can also look here for a list of possible node types 🤯.)

Visitor: Cops

Each linting or formatting rule in Rubocop is encapsulated in a “cop”, which is essentially a visitor. For example, let’s look at the DefEndAlignment cop. By default, this checks that end is aligned to def at the start of the line, e.g.:

# bad
def foo
    end

# good
def foo
end

Each cop defines on_#{node_type} methods, which the traversal mechanism will call when it hits a corresponding AST node. We can see the cop handles 3 node types, on_def, on_defs and on_send:

def on_def(node)
  # ...
end
alias on_defs on_def

def on_send(node)
  # ...
end

(on_send might seem out-of-place - it’s for handling “def modifiers” e.g. private in private def foo.)

If you recall the AST from before, there weren’t any end nodes, and the structure itself left out indentations and spaces. So how does this linter work? Turns out that parser nodes still contain information about the source code (e.g. positions), and cops can use this in tandem with the original source code for linting.

Of note: this cop also does it’s own little traversal for on_send. This is because the send node contains the start position of the private keyword, but the position of end is still in the def node. Thus, the code accesses descendants to look for the actual def node. A visual from our previous example:

Also note that unlike the classical implementation, cops do not implement methods for all node types. Instead, there’s logic that skips non-existent methods during traversal, which is a nice segue to our next section.

Execution & Traversal

Object structure and visitors meet in the Rubocop::Commissioner#investigate method. This method walks the tree and for each node, executes the respective on_#{node_type} method on all enabled cops.

The actual implementation is a little convoluted, with a fair amount of metaprogramming. It’s a fun detour, but feel free to skip this! First, my attempt to visualise the flow with a sequence diagram:

when the Commissioner class is loaded, it loops through node types, defining “original” on_#{node_type} methods via class_eval (source).
#investigate will execute #walk (from an included Rubocop::AST::Traversal module) on the root node of the AST, which calls the corresponding “original” on_#{node_type} methods defined above.
the “original” on_#{node_type} methods will execute every relevant cop for the node.
the “original” on_#{node_type} methods will also execute super, which calls an “inherited” on_#{node_type} method (again defined in Rubocop::AST::Traversal via metaprogramming, see an example for on_send).
the “inherited” method loops through child nodes, again invoking the original on_#{node_type} methods in Commissioner, this time with the child nodes.

I haven’t dug up the why behind this particular design, my guess so far is (1) metaprogramming makes it easier to support new node types, and (2) this was a way to separate the traversal logic from the cop execution logic.

Note that the classical visitor implementation doesn’t dictate where traversal logic should go. In Rubocop, the main traversal is separate from both object structure and visitors, but in other situations it might work better to put the logic in the object structure (or some parts of it, e.g. composite elements might encapsulate their traversal) or in the visitor itself (e.g. if different visitors need to traverse in a different order)³.

Tradeoffs

Now that we know how Rubocop implements a visitor-inspired pattern, let’s briefly examine the benefits and cost of the design.

Benefit: Encapsulation of each Linting Rule in a Visitor

Each cop encapsulates the linting and formatting logic for a single rule, making it simple to disable or restrict the scope of that cop, without affecting other cops. In Rubocop, you can disable cops via a small config change. The traversal logic will then ignore those cops.

Benefit: Ease of extension by adding new Visitors

The object structure and traversal logic is separate from the visitor logic - this makes it easy for a user of the gem (us!) to add and enable new custom cops, without needing to touch the object structure or traversal code.

Cost: Coupling to Object Structure

Changes to the underlying object structure may ripple through all visitors. For example:

If node categorisation changes, cops may no longer be checking correctly for those types e.g. in Ruby 3, how pattern matching was parsed changed slightly.
If the hierarchy changes and ordering of nodes change, it could break traversal. We saw this coupling earlier in DefEndAlignment#on_send, which expects that def nodes are children of send. If that changes, the cop wouldn’t work properly. Arguably, this is rare in Ruby’s case, and can also be mitigated through more abstract traversal (e.g. node pattern, which reminds me a lot of CSS selectors).

Cost: Reduced encapsulation of Object Structure

Depending on the needs of visitors, objects in the structure may need to expose more information, potentially breaking encapsulation. In our example, Rubocop cops need to know about positions in the actual source code through methods like #loc, which is less useful for a program emulating the Ruby interpreter - it only needs to execute the logic represented by the node types.

Afterword

I hope this helped concretise when and why you might want to use the visitor pattern. Here are some other examples you may want to look into:

nokogiri, a XML/HTML parsing library, has a CSS::XPathVisitor.
syntax-tree, another Ruby parser, has visitor support (see PR)
graphql-ruby, a GraphQL parser, also has an AST Visitor

I was invited to share about this at RubySG, here’s a video with some cold jokes:

See Wikipedia.↩
See class diagram in Visitor chapter of Design Patterns book, or on refactoring.guru.↩
See “Who is responsible for traversing the object structure?” in Visitor chapter, Design Patterns book ↩

Overspecifying Tests - the Employee Report Kata

Wed, 22 Jun 2022 00:00:00 GMT

I paired recently with @andyw8 on the Employee Report kata¹, which illustrates how tests become more difficult to maintain if they are “overspecified”. For such an unassuming kata, we found a lot to discuss and ponder about. I found his refactoring particularly interesting and wanted to share it.

What is Overspecifying?

In the context of the kata, a test is overspecified when it unintentionally (or intentionally?) tests more than it should. This couples the test to those additional behaviours, causing the test to fail if those behaviours change. Walking through the kata gives us some great examples, so let’s learn while doing!

The Kata

The instructions are to build a program for a shop owner. The program should generate a report of employees allowed to work on Sundays, given input data like the following:

employees = [
  { name: 'Max', age: 17 },
  { name: 'Sepp', age: 18 },
  { name: 'Nina', age: 15 },
  { name: 'Mike', age: 51 }
]

There are 4 requirements, and you’re encouraged not to look ahead, instead testing and implementing one requirement at a time. We’ll use Ruby and Minitest in our example.

💡 Suggestion: Before going further, you may want to try the kata for yourself - it’s pretty short! When I try katas unprompted and then look through someone else’s solution, I either learn more from their solution or can take a more critical viewpoint.

First Requirement: 18 and above

We first want to exclude employees <18 years of age. This is our first test:

class EmployeeReportTest < Minitest::Test
  def test_return_employees_older_than_18
    employees = [
      { name: 'Max', age: 17 }, # exclude
      { name: 'Sepp', age: 18 },
      { name: 'Nina', age: 15 }, # exclude
      { name: 'Mike', age: 51 }
    ]
    result = EmployeeReport.new(employees).sunday_allowed_employees
  
    assert_equal ['Sepp', 'Mike'], result.map { |employee| employee[:name] }
  end
end

We then write the code to get the test passing ✅. (I’ll mostly link out the implementations because I think it’s secondary to the point of the kata.)

Second Requirement: Sorted Names

The second requirement is to sort employees by name. We write another failing test:

class EmployeeReportTest < Minitest::Test
  def test_return_employees_older_than_18
    # ...
  end

  def test_result_sorted_by_name
    # include only employees that are >18 to keep the test focused.
    employees = [
      { name: 'Sepp', age: 18 },
      { name: 'Mike', age: 51 }
    ]
    result = EmployeeReport.new(employees).sunday_allowed_employees

    # the only difference from the previous test is the order of the result.
    # the requirement wasn't specific, so we default to alphabetical order. 
    assert_equal ['Mike', 'Sepp'], result.map { |employee| employee[:name] }
  end
end

We do a small refactor before addressing the requirement. Our new test passes! ✅

One problem: when we run all the tests, the first test fails ❌. This is in spite of the logic being accurate (our results still excludes employees <18 years). It’s clear that in addition to checking that employees were filtered out, our first test also implicitly tested the ordering.

RSpec has a matcher called contain_exactly that disregards the ordering. Minitest does not, but we could easily sort the expected and actual results so that it disregards ordering:

class EmployeeReportTest < Minitest::Test
  def test_return_employees_older_than_18
    # ...
  
    result_names = result.map { |employee| employee[:name] }
    assert_equal ['Sepp', 'Mike'].sort, result_names.sort
  end

  def test_result_sorted_by_name
    # ...
  end
end

(EDIT: our previous implementation used assert/refute_includes to convey the intent that we’re checking only for the presence of certain employees. @deliberatecoder correctly pointed out that this would not catch duplicate employees. I think this new implementation still captures the intent of the test, which is that we return only employees >18 years old.)

Now the first test is decoupled from the ordering of the result, but there is still at least one hidden coupling…

Third Requirement: All Caps

Now, we are asked to uppercase all the names:

class EmployeeReportTest < Minitest::Test
  def test_return_employees_older_than_18
    # ...
  end

  def test_result_sorted_by_name
    # ...
  end

  def test_results_are_uppercased
    employees = [
      { name: 'Sepp', age: 18 },
      { name: 'Mike', age: 51 }
    ]
    result = EmployeeReport.new(employees).sunday_allowed_employees

    # OK, you got us, we are potentially overspecifying here too 😝.
    assert_equal ['MIKE', 'SEPP'], result.map { |employee| employee[:name] }
  end 
end

We naively implement the third requirement with some ugly code. Our new test passes ✅, but guess what? Our first two tests are now failing ❌. Turns out they were also implicitly testing that an employee’s attribute was in a certain format, when all we actually wanted to check was that the employee itself was included (properties VS identity?).

Optional: Refactoring to Objects

Now we could simply change all the tests to check for capitalized names, but I thought the refactoring @andyw8 proposed was very cool (though you can poke holes in it, see the afterword). Could we get the tests to compare the “identity” of the employees, instead of one of their attributes?

What we did was to introduce a new Employee class and use that during comparisons, instead of checking for an attribute e.g.:

class EmployeeReportTest < Minitest::Test
  def test_return_employees_older_than_18
    employees = [
      # Note the new Employee class.
      max = Employee.new(name: 'Max', age: 17),
      sepp = Employee.new(name: 'Sepp', age: 18),
      mike = Employee.new(name: 'Mike', age: 51)
    ]
    result = EmployeeReport.new(employees).sunday_allowed_employees
    # We're now comparing the Employee objects,
    # instead of their attributes.
    assert_contains_exactly [sepp, mike], result
  end
  
  def test_result_sorted_by_name
    employees = [
      sepp = Employee.new(name: 'Sepp', age: 18),
      mike = Employee.new(name: 'Mike', age: 51)
    ]
    result = EmployeeReport.new(employees).sunday_allowed_employees
    # Same here - this test doesn't rely on the 
    # attributes of an employee!
    assert_equal [mike, sepp], result
  end

  def test_results_are_uppercased
    employees = [Employee.new(name: 'Sepp', age: 18)]

    result = EmployeeReport.new(employees).sunday_allowed_employees
    assert_equal result.first.name, 'SEPP'
  end

  private

  def assert_contains_exactly(expected, actual)
    assert_equal(expected.sort, result.sort)
  end
end

Note that some liberties were taken since the requirements gave us a lot of autonomy: we assumed that we had some way to convert the input (an array of hashes) into objects, and that the output could also be a set of objects.

In the context of the kata, this refactoring is optional because the way the final requirement is written doesn’t require it. But it’s a really good example of how we can encapsulate logic in objects! I did have to take quite a few steps (you can follow the commits here) to “safely” refactor, but the end result is quite satisfying.

Fourth Requirement: Sort Descending

We now want the names to be in descending order. Since we already have a test for ordering, we can modify that instead of creating a new one:

class EmployeeReportTest < Minitest::Test
  def test_return_employees_older_than_18
    # ...
  end

  def test_result_sorted_by_names_descending
    # ...

    # swap the ordering
    assert_equal [sepp, mike], result
  end

  def test_results_are_uppercased
    # ...
  end
end

With the newly refactored Employee class, we can modify the comparison method <=> (a.k.a. the “spaceship” operator). This time, none of our other tests fail ✅ - we have successfully defeated overspecification!

Or have we?

Takeaways

I don’t think most of us intentionally overspecify our tests, so this kata is more a reminder to compare what we’re actually testing VS what we want to test. Could we be more specific?

From the kata, some possible heuristics (we all know that heuristics don’t always work, right?):

when checking for the presence of an object in a result, avoid implicitly comparing the ordering.
when comparing objects, avoid comparing by their attributes and try to find some way to compare their identity instead.

Afterword

I had a lot of fun doing this kata, and @andyw8’s refactoring reminded me of why I appreciate object-oriented programming. That said, there are some decisions made in the refactoring that can be debated or improved:

Employee#name always returns the capitalized name. This might not be an assumption we want to apply application-wide (e.g. what if we wanted another report that only capitalizes the initials), and could be considered too pre-emptive.
Employee#<=> could be similarly argued, though IMO this seems more broadly applicable across an application as a “default” sort.
In our tests, we are relying on comparison of the Ruby object ids, meaning that the EmployeeReport class must return the exact instances that were passed into it. If some step in between created new instances (e.g. for immutability), these tests would fail. (I think this can be easily resolved through adding Employee#== though.)

For the first two points, we won’t truly know if the refactor was “right” or “wrong” until we get new requirements! So, observe future changes, and continue to refactor if necessary.

There are also some decisions that could be worth expounding on… another time:

not memoizing the array of employee data, instead choosing to “duplicate” for each test (eventually we purposefully changed the data for each test).
choosing to have many focused tests, when perhaps one big test might have achieved a similar result.

What is a kata? The term has origins in martial arts, but in software it seems to generally refer to practice exercises. I like the parallel of following and repeating a certain set of movements, but I don’t think everyone subscribes to this specific interpretation.↩

Coupling is a Tradeoff

Wed, 08 Jun 2022 00:00:00 GMT

In software, coupling is often described as something intrinsic and fixed: code is either coupled or not. I’ve had this viewpoint challenged recently during a talk by Kent Beck, where he makes this statement (~14:00):

Coupling is always with respect to a particular change

This formulation suggests we stop treating coupling as something that is strictly intrinsic to our code. Rather, we consciously (or sub-consciously) make coupling tradeoffs when choosing how to modularize our code.

What does this mean for us? I had these takeaways from the talk:

Not all coupling is obvious from the code - a new change may reveal hidden coupling. Thus, observation should be an important part of developing new features.
Coupling is difficult to completely eliminate from a large system, YAGNI¹ is a good perspective when making coupling tradeoffs.
Refactoring is an important part of the toolbox - it helps us change our coupling tradeoffs.

Before we look at each point in detail, let’s align on our understanding of coupling.

What is Coupling?

Beck gives a pretty simple definition of coupling: given two software “elements” (this could be at multiple levels, from code modules to web services) and a requested change, the elements are coupled if both need to be modified to accomodate the change.

I like how this hints at why decoupling is important - if we need to modify many elements to effect a change, that change is likely to be more costly. So a key advantage of decoupling is potentially reducing future cost.

An Example: Extracting Twitter’s “Algorithm”

In a blog post, Beth Andres-Beck shares a great example of how coupling varies depending on the change required. Recently, Elon Musk mused about extracting and open-sourcing “Twitter’s algorithm”, and this is apparently no small feat. Beth observes that the difficulty is likely due to “the algorithm” being spread across multiple microservices.

Note that this doesn’t mean that Twitter’s services are poorly designed - in fact, Beth argues:

With respect to different changes, Twitter’s system is already highly cohesive and loosely coupled. Twitter grew revenue 16% yoy last quarter. It has recently made obvious strides in reducing harassment & abuse on the platform. All of that involved hundreds of engineers safely evolving running software.

To illustrate, let’s assume multiple microservices are involved in determining a user’s feed:

In daily work, a dev working on Service A would have been able to make modifications without touching other services. From that perspective, the services were loosely coupled.

However, to extract out “the algorithm”, all the services would need to be modified i.e. from this perspective, they are coupled.

Not all coupling is discernible from the code

In Kent Beck’s talk, he also mentions that the most expensive couplings are the ones “you don’t see”. His example is somewhat extreme: a change in the backup schedule of an application blocked connections and starved other applications on the same server rack.

John Ousterhout² gives a more relatable example - changing the background colours in a website. If the colours were defined in an inline style on each page, a developer would need to modify all pages. Yet, this coupling (Ousterhout refers to this as “complexity”) isn’t obvious from characteristics of the code itself - there aren’t any shared dependencies.

My takeaway? When there are new requirements or changes needed, observe which code needs to be modified. Do you need to modify code in multiple modules or services? That is a sign that they are coupled with respect to that change, and may need to be refactored to make those modifications easier. In the background colour example, this might mean some central variable to control the colour (you could argue this introduces a coupling to that central stylesheet).

Coupling is difficult to completely eliminate from a large system

Following on from the server rack example, Kent Beck emphasises that the bigger and more complex the system, the more likely you’d find “weird, action at a distance coupling”.

Even if we break our logic into small, composable parts, those parts need to be combined and orchestrated. This gets repeated at multiple levels in software (from modules to services etc.), creating coupling based on how those parts are composed.

The Twitter example shows that a choice was made to decouple a certain “axis”, spreading the generation of a user’s feed across multiple services. This allowed the company to iterate and evolve quickly, with the tradeoff that it was difficult to extract and isolate the algorithm as a whole.

So perhaps it’s worth thinking of coupling as a tradeoff, rather than a fixed property. We decide certain changes are more important to enable today and decouple along those axes, knowing the result could still be coupled from certain perspectives.

Refactoring is important

Earlier, I chose to emphasise observation of changes over speculation, because it’s very easy to get it wrong and cause more harm than good. It’s more important to be able to adapt, and that’s why it’s important to get good at refactoring. To quote Kent Beck from elsewhere:

You might be able to guess that I care less about predicting the future than in doing a good job with the present and adapting flexibly to learning and surprises.

I’ll lean on Beth Andres-Beck’s example again, where he too invokes YAGNI (you ain’t gonna need it):

Imagine that Twitter had guessed that a billionaire would get mad about an ad they showed him. They might have spent a similar amount of time & effort as this project will take today making the ad targeting logic cohesive & decoupled. The code still wouldn’t be any more cohesive or loosely coupled with regards to the change that [Elon Musk] actually wants. It would have cost a bunch of money to do, making all the other work over the years harder, and it still wouldn’t make this change any easier.

Attempting to anticipate the future doesn’t help us build systems that can adapt to it.

So, prioritise getting good at (1) observing the system to identify coupling and (2) knowing how to refactor safely, rather than getting better at speculating future requirements. Refactoring should be seen as an essential part of our toolbox for attaining “low coupling and high cohesion”.

(Note that the above refers to guessing future requirements, as opposed to the requirements themselves being speculative. I think the latter is somewhat true most of the time - you don’t truly know if a feature is going to be a success until it’s out in the wild.)

Afterword

I only touched on coupling in this post, but Kent Beck also talks about cohesion in his talk - it’s a good watch, though you may want to skip the first 9 minutes of banter. Here’s a good timestamp to start from.

Personally, I hope to revisit this post with more concrete, personal examples. Also, few things are truly black-and-white in software design - let’s see if I still agree with these takeaways or develop more nuance as time progresses.

In the meantime, I’d love to hear your thoughts and discuss further - feel free to tweet or leave a comment!

You Aren’t Gonna Need It, an acronym coined by Kent Beck. Here’s a detailed explanation by Martin Fowler.↩
A Philosophy of Software Design, Chapter 2.3↩

Zettelkasten from "How to take Smart Notes"

Sat, 16 Apr 2022 00:00:00 GMT

If you’re trying to get more productive at writing or content creation (especially long-form), I highly recommend checking out How to take Smart Notes. It’s a book by Soenke Ahrens which describes “Zettelkasten”, a system of note-taking and personal knowledge management geared towards writing. Here’s a review and summary of the book, which I think would be more accurately titled “Note-taking to generate ideas for Writing”.

Key Takeaway: Purposeful Note-taking

To date, I’d been focused on the mechanics of Zettelkasten without understanding it’s goals and philosophies. My note-taking was an attempt to make it easier to find and retain information - in effect, creating a personal wiki. So this passage from the book caught my eye:

…it would be rather misleading to think of his slip-box as a personal Wikipedia or a database on paper. The similarities are obviously there, but the subtle differences are what makes this system unique.

Note-taking, in Ahren’s opinion, should be more purposeful. In a Zettelkasten system, the resultant notes should lead to publishable content:

Instead of focusing on the in-between steps and trying to make a science out of underlining systems, reading techniques or excerpt writing, everything is streamlined towards one thing only: insight that can be published.

Eventually, you should be able to get a “critical mass” of notes, such that they become an “idea generator” for your writing:

There is one reliable sign if you managed to structure your workflow according to the fact that writing is not a linear process, but a circular one: the problem of finding a topic is replaced by the problem of having too many topics to write about.

How can our notes achieve this? I like to think of it as building an “idea bank” 🏦, where notes are already processed, potential ideas for future writing. They should also be connected to other relevant notes - this allows us to follow the chain of notes to find related ideas (I like to imagine this as creating literal “trains of thought” 🚂).

I’ve attempted to summarise the process below. Important: The process, while a little abstract, is surprisingly simple. The catch is it depends a lot on you actually processing what you read e.g. rephrasing to develop understanding, connecting the ideas to existing ideas.

Summarising Zettelkasten

“Zettelkasten” means “slip-box” in German. This is in reference to the physical approach of writing on index cards and filing them in a box. Despite the physical roots, it’s still very applicable in today’s digital environment, and can be summarised as follows:

Take “Fleeting Notes” or “Literature Notes”
Write “Permanent Notes”
Connect and Index your Permanent Notes
Expand upon ideas in your Permanent Notes
Write from your ideas

1a. Take Fleeting Notes

These are effectively shower thoughts 🚿, ideas that come to you in random places. Have a way to quickly write them down, and don’t overthink it - they’ll get discarded once processed into Permanent Notes (Step 2) anyway. Note: this isn’t a rigid process - if a thought is already well-formed, it can become a Permanent Note straightaway.

1b. Take Literature Notes

These are notes that you take when reading something, the primary goal being to develop understanding of the text, which will help later in the process. Avoid copying directly - rephrasing helps you develop and check your understanding. Although Ahrens says you are free to use any note-taking methodology or technique, he also suggests to be picky. Since the purpose isn’t to remember everything (you can add a bibliographical reference for that), ideally the notes taken should be concise and relevant to you.

For example, Ahrens read a book about scarcity and how people seem to react irrationally to it (e.g. people with little money spend unnecessarily on luxuries like take-away food):

I took some literature notes collecting reasons how and why humans act so very differently when they experience scarcity. This was step one, done with an eye towards the argument of the book. I had questions in mind like: Is this convincing? What methods do they use? Which of the references are familiar?

Above, we can see that his focus was on understanding the book’s argument (the reasons why humans act differently). The questions at this point are narrower, focused on the text itself.

2. Write Permanent Notes

Here we begin to process the notes in Step 1, converting them into “permanent notes”. This involves thinking about how they relate to your research and interests (which comparing to existing Permanent Notes can help with):

The idea is not to collect, but to develop ideas, arguments and discussions. Does the new information contradict, correct, support or add to what you already have (in the slip-box or on your mind)? Can you combine ideas to generate something new? What questions are triggered by them?

This is perhaps the most abstract step of the process - how do you know what is worth keeping, especially in the beginning when you have few notes to compare with? Though he doesn’t explicitly say this, Ahrens talks about “what interests you” a lot, and I think that’s a decent heuristic - write about what you find “interesting”. If it interests you, there’s a higher chance you’d want to expand on it in the future.

Another suggested approach for writing Permanent Notes is to write about why something seemed important to you. Following on the previous Literature Note example in Step 1b - since Ahrens was interested in “a theory of society” (🤷‍♂️), his Permanent Note was:

“Any comprehensive analysis of social inequality must include the cognitive effects of scarcity. Cf. Mullainathan and Shafir 2013.”

Still pretty vague? Unfortunately, yes (at least to me). There is a further example in the book’s appendix which I’ll attempt to break down in a future post.

3. Connect and Index Permanent Notes

The next step is to file/store your Permanent Notes, making sure to add links to and/or from other relevant notes. This will help us find related ideas later when writing.

To further aid discoverability, keep an “index” of topics or keywords, and make sure the new Permanent Note can be reached from the index. This can be done by adding a link to the note directly next to the topic, or linking it from another note that can be reached through the index.

The above index might seem irrelevant given the searching functionality in digital solutions, but Ahrens suggests our keywords be more situational instead of purely for categorisation (“In which circumstances will I want to stumble upon this note, even if I forget about it?”). His example was the following note:

“Tversky/Kahneman (1973) showed in an experiment that people are more likely to overestimate the likelihood of an event to happen if they are able to conceive it well and in detail than if it were abstract.”

If we took a categorisational approach, the keyword could be “misjudgements” or “experimental psychology”, which he doubts could be turned into a structured argument. Instead, we should look through existing notes and think about trains of thought and questions.

If the writer was an economist working on decision-making, he or she might be thinking about how management prefers projects with easily-visible outcomes, even if they’re less profitable. A good keyword could then be “capital allocation problems”.

As a topic develops, you may want to add an “Overview Note” - a note which starts to structure thoughts and links to relevant Permanent Notes. Unfortunately the book doesn’t give a clearer example of this, though Ahrens says it could be an in-between step to the development of a manuscript (e.g. laying out the arguments for the topic).

4. Develop ideas in Permanent Notes

Ahrens talks about developing topics, questions and projects “bottom up” from the system. To my understanding, this is more about mindset than a mechanical difference from Step 3 - instead of starting “top down” like brainstorming topics to write about, look into your notes and expand upon what looks promising (e.g. where clusters of notes form).

What you end up writing about may be very different from what originally interested you! I love his example about how DNA’s structure was discovered - the original research project was to find a treatment for cancer. As they researched, the scientists were able to pivot to the most promising path.

5. Write from your ideas

When an topic is developed far enough, you can copy notes to an outliner and re-order them. This helps you identify what’s missing for further research and note-taking, as well as remove redundant information. At this stage, it’s also easy to try out different ideas and structures for articles. Then… write!

The Review

✅ This book gets the wasabi stamp of approval, especially since I’m trying to get better at writing. If the summary above intrigued you, please give the book a read! There’s a lot of detail and nuance that I didn’t cover (and honestly, still trying to process how it applies to me). One point worth re-emphasis - the system summarised above is just a skeleton. Processing what you read and thinking about how it connects is what’s actually important.

Here’s some other thoughts:

Time Commitment: Low. Amazon says that this book takes just above 4 hours to read. Personally, I took longer for a first read-through, but was still able to finish it with a few days of intermittent reading (probably < 10 hours).
Depth: I think the content is pretty dense, in a good way. There’s a lot to ponder about. I like how Tiago Forte puts it: “Every paragraph has a point, and I struggled to leave anything out of this summary.” However, I did feel that the book was more abstract and spoke mostly in terms of principles. I could have used more concrete examples to visualise and understand the process.
Writing Style: You can see the influence of academia in citations and references to other papers, but I thought overall the writing was simple to read. The words weren’t unnecessarily flowerly or complex.

Why use Polymorphism in OOP?

Sun, 20 Mar 2022 00:00:00 GMT

When I first came across the concept of Polymorphism in Object-Oriented Programming (OOP), it wasn’t obvious to me why it was important.

Let’s start with a hot take by Robert C. Martin:

…the thing that truly differentiates OO programs from non-OO programs is polymorphism.

It’s a bit of a blanket statement (Martin states that this was purposefully reductive), but I do agree that polymorphism is a central aspect of OOP. Let’s dig deeper!

What is polymorphism?

Since polymorphism has different meanings depending on context¹, let’s align on what I meant by “Polymorphism in OOP”. In the earlier blog post, Martin explains it as different objects being able to accept the same message, implementing their own behaviour. I’ll paraphase his example:

some_object.do_something()

We don’t actually know what some_object is! Nor does it actually matter. Many different implementations could replace some_object, and so long as they have the same interface (i.e. have the method #do_something), the program would still run.

Why use polymorphism?

In another blog post, Martin says:

There really is only one benefit to Polymorphism; but it’s a big one. It is the inversion of source code and run time dependencies.

It wasn’t initially obvious to me what that meant, so let’s concretise this with an example: caching in a web framework like Rails. There are multiple options out there for caches - for example, Rails supports a FileStore, a MemCacheStore and a RedisCacheStore. For the sake of illustration, imagine that the caches all had their own interfaces:

def example_method
  # do stuff
  if Rails.cache.is_a? NaiveFileStore
    data = Rails.cache.search_for(key)
  elsif Rails.cache.is_a? NaiveRedisCacheStore
    data = Rails.cache.get(key)
  end
  # do more stuff
end

Above, our example_method has a direct dependency on NaiveFileStore and NaiveRedisCacheStore. If their interfaces change, or new kinds of store need to be supported, it will require changes in example_method.

Earlier, Martin talked about an inversion of dependencies (the “D” in SOLID) - practically, this means example_method and the different cache stores agree on an interface, and both depend on it. This makes all cache stores polymorphic! In Rails, all cache stores implement a #fetch method, so the above code becomes:

def example_method
  # do stuff
  data = Rails.cache.fetch(key)
  # do more stuff
end

The code in example_method is more concise now, but that’s a side benefit; what’s important is it’s shielded from knowing which cache store is used, and how the store works. It just knows that some cache store exists (Rails.cache), and the cache store has agreed to implement a #fetch method.

We can visualise this “inversion” below - the arrows can be thought of as “dependency directions”:

Martin describes this as a “plugin architecture”:

This inversion allows the called module to act like a plugin. Indeed, this is how all plugins work… Plugin architectures are very robust because stable high value business rules can be kept from depending upon volatile low value modules such as user interfaces and databases.

With the new setup, the polymorphic cache stores can be swapped for each other without changing the code in example_method. New kinds of cache stores can also be supported by creating classes that implement #fetch and other methods as specified by ActiveSupport::Cache::Store.

In short, polymorphism makes it easier to extend or change aspects of our programs, without a rippling of changes throughout the entire program. (How do we determine the “aspects” to split our programs by? A good read would be Parnas’ classic 1972 paper.)

Afterword

While our cache example does rely on inheritance to make the polymorphic group explicit, inheritance is not a prerequisite. For example, each cache could be a completely unrelated class, and it still would work in Ruby as long as they had a #fetch method (this is also called “duck-typing”).

You may have spotted that Martin talks about inversion of “source code” in addition to runtime dependencies. I understand this has greater implications in compiled programs, but am not familiar enough to expound on it. Do leave a comment if you have an example!

There are many other neat examples relating to polymorphism. Here’s some I thought of:

Martin Fowler’s refactoring book has a “Replace Conditional with Polymorphism” refactoring, which I think was neatly illustrated in this Sandi Metz talk.
Many Behavioural Design Patterns rely on polymorphism, for example the Strategy, Command, Visitor etc.
The NullObject is a neat way to support a null value or no-ops when your code relies on a Polymorphic interface. (Note that while Rails has a NullCache, it doesn’t solve the same issue that the pattern is meant to.)
Ruby-specific example of a “Plugin Architecture”: Rubocop allows you to add your own custom cops without changing the library’s source code, because all cops are polymorphic!

The definition I used for polymorphism is actually somewhat narrow, but seemed like a relatively common understanding in the context of OOP. The Wikipedia article on polymorphism shows a lot more breadth in the topic, and I think what is described above is known as single, dynamic dispatch. There’s some really good discussion on this in the Reddit thread for this post.↩

A 1972 paper and the Single Responsibility Principle

Mon, 28 Feb 2022 00:00:00 GMT

”A class should only have one reason to change” is a mantra that Object-Oriented advocates have chanted for years. Dubbed the “Single Responsibility Principle” (SRP), it remains somewhat abstract till this day ✨. Abstract enough, in fact, that it’s originator (Robert Martin) felt it worthwhile to explain again in a blog post - 14 years after he first wrote about it!

That blog post begins by referencing and quoting a 1972 paper, ”On the Criteria To Be Used in Decomposing Systems into Modules” by David Parnas:

“We have tried to demonstrate by these examples that it is almost always incorrect to begin the decomposition of a system into modules on the basis of a flowchart.

We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others.”

The paper seemed significant, as Martin wrote that the SRP appeared “to align with Parnas’ formulation”. Could it demystify the SRP? What exactly was this paper about?

(Note: I’ve tried not to abstract too much of the paper, in the hope that you’d be able to draw your own conclusions as well.)

The System: KWAC Index

First, Parnas sets the stage - he would compare two criteria for modularizing a system, showing that one provided superior flexibility. The system in question was a KWAC (KeyWord Alongside Context) index generator.

KWAC was an indexing system for technical manuals, allowing a reader to quickly find where in the manual a keyword was used. In addition to showing the keyword, KWAC would show the rest of the sentence as well, it’s “context”:

To build the KWAC index, the system would take in file of sentences. Say the input file looks like this:

# cat input.txt
Program Development # line 1
On The Criteria # line 2

For each line, the system looked through every word (the “keyword”), generating a KWAC entry. This was done by “circularly shifting” words that came before the keyword, appending them to the end. Then, it would sort the entries alphabetically. For example, given the above file, we’d get:

Criteria. On The [2] # "On The Criteria" is the original, "Criteria" was selected as the keyword here
Development. Program [1]
On The Criteria [2]
Program Development [1]
The Criteria. On [2]

The next sections explain Parnas’ modularizations. This video explanation (starting from 17:16) helped me visualise and understand them, and I’m borrowing liberally from it 🙇‍♂️.

Modularization 1: “Flowchart”

In the first modularization, Parnas modelled the problem as a flowchart, using the individual steps to break apart the modules (an approach which I’m guilty of applying at face value):

There’s a design flaw that isn’t obvious in the diagram - some common data that all modules need is stored in memory (a modern parallel might be a datastore), each module needing to know the low-level layout of the data. We’ll contrast this with Modularization 2 later.

Here’s what each module did:

Input
- Parsed the input text file and stores in memory. We’ll call this “Characters”.
- The video (23:36) that explains this as (1.) memory at the time was accessed in groups of 4 characters (bytes), (2.) the input was split across these groups, with spaces in between words and lines.
- The “Characters” could be visualised as follows:
- Note that in the above, line breaks are not distinguished from spaces. The module outputs a separate “Lines” structure for this, which is an array of “starting positions” for each line (e.g. above, Character 1 is the start of line 1, Character 21 is the start of line 2)
Circular Shifter
- Accessing “Characters” in memory and “Lines”, it outputs an array of “Shifts”.
- Each “Shift” stored the line number, as well as the starting index of each word.
Alphabetizer
- Accessing “Characters” and “Shifts”, it sorts the “Shifts” alphabetically and outputs them as “Alphabetized”.
Output
- Accessing “Characters” and “Alphabetized” shifts, outputs the KWAC index (see “The System: KWAC Index”).

A helpful visual summary of all the outputs and stored data from the video explanation:

Modularization 2: “Information Hiding”

In this modularization, Parnas uses the criteria of “Information Hiding”. The definition is worth mentioning for it’s relevance to the SRP:

Every module in the second decomposition is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.

Interestingly, what one might typically think as the general “responsibility” remains the same - modules are still mostly split along the same functionality lines. Instead, the major differences that I noticed in this modularization were:

The details of how the characters were stored were encapsulated (hidden) via a new module, “Line Storage”. This meant that other modules did not have to know the low-level layout of the data. Instead, they would have a simpler interface to store and retrieve characters.
The dependencies of modules were also tweaked, such that only two modules interact directly with “Line Storage”, as opposed to every module having to access “Characters” in Modularization 1.
Modules in Modularization 1 depended on outputs from previous modules or common in-memory data. In contrast, modules in Modularization 2 depended directly on the interface of other modules (similar to calling methods on the module, a more Object-Oriented style).

Each module in greater detail:

Input
- Like Modularization 1, also parses the input file, but this time initializes a “Line Storage” and inserts the data.
Line Storage
- Think of it like an Object that has methods for inserting and retrieving characters by line (e.g. get Character 5 of Line 1, Word 2).
- Note: Unlike Modularization 1, this means callers do not need to know the nitty-gritty details of how data is stored.
Circular Shifter
- Generates the circular shifts from Line Storage in an initialization step.
- Instead of outputting “Shifts”, the module itself provides an interface (like an Object) similar to Line Storage, but allows retrieval of characters by their shifts instead of lines (e.g. get Character 5 of Shift 1, Word 2).
Alphabetizer
- Retrieving the characters from Circular Shifter, this modules sorts by alphabet and remembers the shifts in an initialization step.
- Instead of outputting the “Alphabetized” shifts, the module also provides an interface for getting ordered Circular Shifts indexes (e.g. tell me the shift number that’s in the 2nd sorted position).
Output
- Uses Alphabetizer and Circular Shifter to generate the KWAC Index.

Comparison: Changeability

While other comparisons were made, the analysis of changeability speaks most to the SRP. Parnas begins by suggesting some “likely” change scenarios. Most require many modules to be updated in Modularization 1, but have a much smaller blast radius in Modularization 2.

Let’s take Scenario B below as an example - since all the modules in Modularization 1 depended upon a specific layout of “Characters” in memory, changing that layout would require changing every module. However, in Modularization 2, this would only require a change in the implementation of “Line Storage” - the interface could stay the same, meaning other modules were shielded from this change!

Scenario	Modularization 1	Modularization 2
A. Deciding to store “Characters” in a different medium (e.g. filesystem) instead of memory	All modules	Line Storage
B. Deciding on a different way to store the “Characters” (e.g. group by words instead of every 4 characters)	All Modules	Line Storage
C. Deciding to store the full sentences of each circular shift instead of their indexes	Circular Shifter, Alphabetizer, Output	Circular Shifter
D. Deciding to change “Alphabetized” generation to be lazy or distributed (possibly due to a large dataset?)	Difficult to achieve as computation must be completed before output	Achievable as Output doesn’t need all the shifts to be “alphabetized”

So the takeaway here is that good “information hiding” (a.k.a the SRP) results in a system that adapts more easily to changes.

Conclusion

So where does the paper leave us in relation to the SRP? Personally, I left with a better understanding of why it was important - it should result in a system that is easier to evolve with changes in requirements.

However, even with the (helpful) example, we are left with an abstract criterion of “information-hiding” and starting with a “list of difficult design decisions” or “design decisions which are likely to change”. The SRP, it seems, remains more art than science.

Discuss on Hackernews, or continue the conversation on Twitter.

(I apologise for the mix of American and British english 😆.)

Reading a Ruby gem with VSCode

Tue, 15 Feb 2022 00:00:00 GMT

Whether out of curiosity or trying to understand a method for debugging, it’s helpful to know how to dig into a Ruby gem. Recently I was curious over how mocha, a mocking library for minitest, allowed stubbing on any_instance of a Class (as opposed to injecting a stubbed object), for example:

def test_stubbing_an_instance_method_on_all_instances_of_a_class
  Product.any_instance.stubs(:name).returns('stubbed_name')
  product = Product.new
  assert_equal 'stubbed_name', product.name
end

Following this are steps you can try when investigating a gem. I’d love to say these were the exact steps I took while investigating any_instance, but truth is some steps I discovered only while writing this post 😅. The steps are specific to using VSCode.

Open the Source Code

Assuming the gem is installed, gem open <gemname> (or bundle open <gemname> if installed in a project via Bundler) will open it’s source code with the command set in the $EDITOR environment variable.

For example EDITOR=code bundle open mocha from my project opens the gem in VSCode:

If the gem isn’t installed, you could also look up the source code online (rubygems.org) and if the code is on GitHub, there’s a handy shortcut to open it in a browser version of VSCode.

Find your Target

One of the easiest ways to find the method definition is to use #source_location. Fire up an irb console:

require 'minitest'
require 'mocha/minitest'
Object.method(:any_instance).source_location
# => ["/.../ruby/3.0.3/gems/mocha-1.13.0/lib/mocha/class_methods.rb", 45]

In case that doesn’t work, since we have the source code open, we can make use of VSCode’s search:

Some ideas to try if the results list is really long:

prefixing the search term with the declaration syntax (e.g. class <ClassName> for a class, def <method_name> for an instance method) generally works well, but not always, given the many ways to metaprogram in Ruby. For example, it’s not obvious that any_instance would have been defined as an instance method.
”Match Whole Word” (ab) helps exclude classes / methods with the search term as a substring (e.g. it would have excluded any_instance_method and mock_impersonating_any_instance_of in the results above)
”files to include” knowing the structure of most gems (see later section), we can usually restrict to the lib/ directory, which will exclude test files, bin scripts etc

Trace the Callers

While we found the method, it didn’t answer our initial question, which was how any_instance becomes callable on any class. We could probably try a “top-down” or “bottom-up” search here.

For “bottom-up” search, since this is plain Ruby as opposed to Rails, we could try to follow the require trail. But if we have an extension like solargraph, a simpler way is to make use of VSCode’s Go to References:

Above, we find that Mocha::ClassMethods is included into Class in Mocha::API:

Class.send(:include, ClassMethods)

We can continue doing the same approach, but at some point the “search tree” got quite unwieldy so I switched to a more “top-down” approach, checking the README to see how the gem was require-d.

Know the Structure of a Gem

A quick detour on how a gem is structured - lib/ generally contains the gem’s core logic. There you’ll find a ruby file and a folder corresponding to the name of the gem. For example, in mocha:

lib/
└── mocha.rb
└── mocha/

The ruby file (e.g. lib/mocha.rb) generally contains most of the gem’s bootstrapping logic, as this is the file loaded when requiring the gem (i.e. require 'mocha' would load lib/mocha.rb). So typically we’d start our “top-down” search here.

mocha is slightly different - it supports multiple test libraries, and in the README we are instructed to require a different file depending on our test library. For example, for minitest we’d do:

require 'minitest/unit'
require 'mocha/minitest'

So in this case we’d start our search from lib/mocha/minitest.rb. There on it’s pretty straightforward Ruby, leading us back to Mocha::API (albeit with knowledge of how it gets required!).

Footnotes

How to Dispel Ruby Magic and Understand Your Gems (Justin Weiss) - an inspiration for much of this
Structure of a Gem (RubyGems) - more details on how a gem is structured, along with a quick start so you can build your own gem!
VSCode docs on searching across files

Practical Metaprogramming in Ruby: minitest/mock

Sat, 22 Jan 2022 00:00:00 GMT

minitest/mock (repo) is a tiny mocking library (< 250 lines of code, including comments) that’s also a great showcase for how metaprogramming can be used in Ruby - let’s take a look! If you’re new-ish to Ruby, I hope this gives you concrete examples of what metaprogramming is possible, and how to do it.

What is metaprogramming?

In the context of this article, I’m referring to programs and code that are able to do introspection (inspect themselves e.g. type or properties) and reflection (modify their behaviour).

First example: Set expectations with method_missing

The main feature of minitest/mock is the Minitest::Mock class, which can be used to test that a method is called, for example:

require 'minitest/mock'

class ClassUnderTest
  def initialize(foo)
    @foo = foo
  end

  def call
    @foo.bar
  end
end

# create an instance of Mock and set an expectation
# that the method `bar` will be called on it
mock_foo = Minitest::Mock.new
mock_foo.expect(:bar, 'example_return_value')

# inject the mock into the Object-Under-Test
# which essentially calls `mock_foo.bar`
ClassUnderTest.new(mock_foo).call # => returns 'example_return_value'

# check that `bar` was called
mock_foo.verify

Notice that there’s nowhere in the library where the method bar is explicitly defined, yet running the above doesn’t raise a NoMethodError. We were able “define” (well, not exactly, but we’ll get to that) a new method after the mock object was already instantiated, via the expect call.

The truth is expect (source code) is straightforward Ruby; it saves the expected call to an Array instance variable - no new method gets defined. The real magic happens in method_missing :

# lib/minitest/mock.rb#L118
def method_missing sym, *args, &block
  # ...
  expected_call = @expected_calls[sym][index]
  # ...
  expected_args, retval, val_block = expected_call.values_at(:args, :retval, :block)
  # ...
  @actual_calls[sym] << {
    :retval => retval,
    :args => zipped_args.map! { |mod, a| mod === a ? mod : a },
  }
end

If a method is called that doesn’t exist, Ruby will invoke method_missing with a few arguments:

the method’s name in symbol form (sym in the above example)
the arguments (*args ) and block (&block) passed to the non-existent method

Minitest::Mock uses this to catch any undefined method that is called, checking it against the saved expectations. This makes it seem like the object has that method defined!

Remove inherited methods via undef_method

While Minitest::Mock doesn’t explicitly define many methods, it does inherit methods from Ruby’s Object class, such as #nil?.

What if we want to check that a mock receives a method call, like #nil?? Without any intervention, method_missing would not be invoked, since those methods exist! Something like this would happen:

mock.expects(:nil?, true)
mock.nil? # => false (this is the default for Object!)
mock.verify # raises MockExpectationError, because the original `#nil?` method is called instead of `method_missing`!

So what the library does here is to undefine them when the class is interpreted. instance_methods is used to introspect the class for any instance methods, which are removed via undef_method:

# lib/minitest/mock.rb#L26
instance_methods.each do |m|
  undef_method m unless overridden_methods.include?(m.to_s) || m =~ /^__/
end

Calling #nil? now would properly invoke method_missing.

Redefining Methods with define_method

In the code above, we saw that not all instance_methods were removed, there are some overridden_methods (e.g. inspect) where the library opts to keep the default implementation unless the user has explicitly set an expectation. This is done by re-defining them via define_method:

# lib/minitest/mock.rb#L30
overridden_methods.map(&:to_sym).each do |method_id|
  define_method method_id do |*args, &b|	
    if @expected_calls.key? method_id then
  	  method_missing(method_id, *args, &b)
    else
  	  super(*args, &b)
    end
  end
end

The goal seems to be so that a Minitest::Mock instance still responds like a Ruby object e.g. when printing it in the terminal 😄.

Stubbing by redefining methods on “Metaclasses”

minitest/mock also has a stubbing feature, which allows you to redefine the return value of an object within a block:

Time.stub(:now, "example") do
  Time.now # => "example"
end
Time.now # => actual time...

Let’s look at the implementation - the library first opens the Object class, and defines a stub method there:

class Object
  # lib/minitest/mock.rb#L214
  def stub name, val_or_callable, *block_args
    # ...
  end
end

Since Object is the ancestor of all almost everything in Ruby (including Classes and Modules), we are able to call stub on basically anything. The actual implementation of stub is the interesting bit, introducing us to Ruby’s “metaclasses” (also known as “singleton classes” or “eigenclasses”):

class Object
  # lib/minitest/mock.rb#L214
  def stub name, val_or_callable, *block_args
  	new_name = "__minitest_stub__#{name}"
	
  	metaclass = class << self; self; end
  	# ...
    metaclass.send :alias_method, new_name, name

  	metaclass.send :define_method, name do |*args, &blk|
	  # return or execute val_or_callable
	end

    # ...
    yield self
  ensure
    # cleanup...
  end
end

There’s some further reading on metaclasses and that mysterious class << self syntax in the footnotes, but one way to understand the above is to replace the following lines:

metaclass = class << self; self; end
metaclass.send :define_method, name do |*args, &blk|
  # ...
end

…with:

def self.name(*args, &blk)
  # ...
end

In other words, stub is defining (or overwriting) the method on the object - the type of method (class/instance) being neatly handled by accessing the metaclass. (Note: self.define_method would not work for class methods, since self returns the class, it would define instance methods.)

The original logic is kept safe by renaming it via alias_method, and in the ensure block, there is code to reverse this change. These ensure stubbing only takes effect in the stub block.

Footnotes

Note that metaprogramming, while powerful, comes with a performance and readability cost. Use with care!

For further reading on various topics touched in this article, check these out:

Mocks aren’t Stubs by Martin Fowler
Notes on Metaprogramming by Ray Toal
Ruby metaclasses:
- Ruby Metaprogramming Is Even Cooler Than It Sounds by Nikola Todorovic
- Diving into Ruby Singleton Classes by Léonard Hetsch
- Metaprogramming in Ruby: It’s All About the Self by Yehuda Katz

Mocking in RSpec VS Minitest: A Cheatsheet

Wed, 05 Jan 2022 23:21:03 GMT

My first exposure to a testing library in Ruby and Rails was RSpec, but I’m now using Minitest. It wasn’t immediately obvious how to translate rspec-expectations and rspec-mocks to Minitest, so here’s my attempt at a cheatsheet:

RSpec	minitest with mocha	minitest with minitest/mocks
`double`	`mock` or `stub`	`Minitest::Mock.new`
`instance_double`	`stub` or `mock.responds_like_instance_of`	Not supported
`expect().to receive().with().and_return()`	`mock.expects().with().returns()` Note: like RSpec, can also call `expects` on any object, not just a `Mock`/`Stub` `with` has imprecise keyword arg matching in Ruby 3	`mock = Minitest::Mock.new` `mock.expect(:method, args, return)` `mock.verify` Note: cannot call `expect` on any Object, but can make a Mock delegate to an underlying object
`expect().not_to receive()`	`mock.expects().never`	Technically you could use `Object#stub` and raise in the `val_or_callable`
`expect()... and yield_control()`	`mock.expects().yields(*parameters)` See docs	Not supported since return value is not a callable
`expect()... and raise_error()`	`mock.expects().raises(error)` See docs	Can pass in a block (instead of expected args) to `expect` and raise there
`allow()...`	Similar as above but substitute `expects` with `stubs`	Can use `Object#stub`
`.and_call_original`	Not supported (see reasoning)	Not supported
`allow_any_instance_of.Klass`	`Klass.any_instance.stubs`	Not supported

It’s very possible I’ve gotten something wrong - please feel free to correct in the comments or submit a PR!

Footnotes

For the curious, I wrote about how minitest/mock works here

My Learning Workflow for 2022

Fri, 31 Dec 2021 00:00:00 GMT

I’ve been working through a course on Coursera recently, and it’s been a great opportunity to reflect on where I am in the “meta” of learning. This post walk throughs my current workflow before diving into the context behind it.

1. The Workflow

The workflow is loosely based on Zettelkasten, with a little thrown in from a “Learning to Learn” course I audited last year. I’m currently using two tools: Obsidian for note-taking, and Anki for testing myself / spaced-repetition. More on these in later sections.

My Obsidian knowledge base is about 6 months old (I’ve yet to migrate my older knowledge base) and looks like this, with a few top level folders:

Articles - where I store my literature notes (explained later). I named it “articles” since it was shorter, which makes it easier to add a note via Obsidian’s quick switcher:
Files - let me state the obvious that files go here, it just makes my root directory cleaner. I don’t bother with subfolders, relying on the backlinks (connections) to find stuff. There’s a useful setting to automatically add all pasted files to this folder:
Blog - is where I record ideas for posts and start drawing connections with the rest of the knowledge base. (Fun aside, this post was brewed 🍺 in Obsidian!)
Templates and Daily Notes are plugin-related, and mostly for what they say (though I still ended up doing most of my journalling in Notion)
The rest of the folders are mostly there from earlier experiments, I will move the stuff out of there… someday 🏃.
All other notes (“permanent” notes) go into the root directory, yay!

And the workflow for this course was:

Start a literature note in Articles for general note-taking
While taking literature notes, create placeholder notes (or link to existing notes) for individual concepts that seem useful or I want to go deeper into:
When I want to internalise something (happens often after failing a weekly assignment 😆), I create a question and sync Anki flashcards via this plugin (having a bit of issues rendering Latex though):
Occasionally I’ll revise the flashcards, either via the Anki desktop app or the web app on mobile (good enough 🤷‍♂️ there are other options)

2. The Context

Next I’ll go deeper into my influences.

Zettelkasten

This is a system of note-taking often mentioned together with Niklas Luhmann, who had a massive knowledge base (some say 90,000 index cards) and credited the system for helping him write prolifically. He wrote more than 70 academic books and nearly 400 scholarly articles! (Update: I did a deeper dive on Zettelkasten here.)

After multiple failed attempts at starting a hierarchical knowledge base, I’ve found elements of this approach freeing. Previously, I often got stymied by categorisation - what title should I give this note so it’s searchable? Which folder should it go into? Should I be using a tag instead?

Now much of this pain is gone, as I eschew traditional hierarchy (deeply nested folders), favouring smaller, “atomic” notes with connections between them (graphs). I think this maps a lot closer to the brain - after all, we draw metaphors or parallels from other disciplines, and may find ourselves often having to recategorise information (especially when learning something new).

A related workflow I’ve adopted is a “pre-processing” step of first taking “literature notes” before converting to “permanent” notes.

Tool of Choice: Obsidian

I first attempted to maintain a Zettelkasten knowledge base in Notion, because:

One-app-to-rule-them-all: I was already using it as a goal setting and project management tool.
It had a mobile app, web clipper and a pleasant UX.
You could (tediously) link various pages together, and they later introduced backlinks.

Ultimately, I moved over to Obsidian, which had less polish but was better in one crucial aspect; it was easier to adopt a Zettelkasten-influenced system:

Building connections between notes was trivial (two square brackets), while it took a few seconds to just to search for or create a new note in Notion.
Searching for a note was much faster, because it’s local first and there is a keyboard shortcut to search by title (Notion defaults to including page text, which slows down search considerably).

What Obsidian also had going for it:

Notes are stored as markdown files; which I thought might be easier to work with if I ever wanted my knowledge base to be public (i.e. a digital garden). My blog also uses markdown files, which made conversion pretty straightforward and automatable.
has a burgeoning plugin system, in particular I wanted to be able to do spaced repetition (see next section).
Local-first (fast, private) and E2E encryption if you pay for their sync service.

Other tools I’ve tried:

Roam - arguably one of the first movers in “graph” knowledge bases, but was clunky when I first tried it plus it got expensive.
Remnote - has built in spaced repetition (see next section) but I found setting up repetition to be a bit unintuitive, and it had no mobile app.

Learning How to Learn (Coursera)

This was a Coursera course I audited last year, and while it felt a bit disjointed, I had a lot of useful takeaways, in particular:

not actively thinking about a problem can actually help you come up with solutions (diffuse thinking)
testing yourself + space repetition is super effective, better than blindly re-reading notes etc.
use memory techniques to help understand and remember: chunking, analogies and metaphors
exercise and sleep help you think better

How I’ve tried to apply this:

learnt to appreciate doing “nothing”, especially when working on a problem requiring creative thinking
wrote questions and used an Obsidian plugin to sync them to Anki (a spaced repetition tool), so I could test myself with spaced repetition. (The “tried” qualifier is important, as in practice this has required quite a lot of effort and discipline 😊)

Growth Mindset

This doesn’t manifest in the workflow or the tools I use, but it’s had a massive influence in my learning journey last year. I had watched short videos on having a growth mindset before, but it was only after reading Carol Dweck’s book that I started internalising it.

Self-doubt has been a constant struggle for me the past few years, and it sometimes manifests as bouts of “why bother if I’m not cut out for it”, i.e. a fixed mindset. Going deeper in the topic via the book has helped me become more aware of my fixed mindsets, and enabled me to action on it. I regret not reading it earlier.

The fixed mindset reared it’s head a few times over the course. At the beginning, I had contemplated giving up after struggling to understand some big-Oh proofs. Most recently, I struggled with self-confidence after failing the weekly problem assignment. Thankfully, I stuck with it, mostly on the thinking of “well, at least I’ll learn something”. There was some personal vindication when I realised I was able to solve all of the questions in the final exam (albeit I still had to take it twice, due to some misreading and miscalculation).

It’s not a perfect book by any means - I’ve found it repetitive and generalising at times - but it’s been more effective at helping me apply the concepts than watching explainer videos or reading articles.

3. Cautious Optimism

My knowledge base is still pretty small, it will be interesting to see whether this still holds up over time, especially as I start taking more courses and learning more things at work. I’m also not very disciplined with regularly going through the base to build connections, but I think that’s also part of it’s beauty - minimal process and mental overheads. Let’s see how the rest of the year pans out!

What’s your learning workflow?

TIL: Extending a Ruby Struct

Sun, 07 Nov 2021 10:48:03 GMT

Last week I had a head-scratching moment looking at the following code (from the google-api-ruby-client):

ClientOptions = Struct.new(
    :application_name,
    :application_version,
    :proxy_url,
    :open_timeout_sec,
    :read_timeout_sec,
    :send_timeout_sec,
    :log_http_requests,
    :transparent_gzip_decompression)

# ...

class ClientOptions
    # ...
    def self.default
    @options ||= ClientOptions.new
    end
end

My initial confusion stemmed from the assumption that the first ClientOptions declaration returned a regular object. A helpful colleague explained to me that Struct.new actually returns a subclass of Struct. That meant the second class ClientOptions declaration was actually re-opening the class to add a method, not to declare a new class.

Why would one want to do this? Well, one would get all the methods of a regular Struct (e.g. accessors, iterators) ~~for free~~ at the cost of the inheritance hierarchy. This could be useful for ValueObjects.

Meta-Postgres: What constraints does a table have?

Sun, 16 May 2021 00:00:00 GMT

TIL that checking the structure of your database is just another query away! I needed to find whether a CHECK constraint existed and learnt that Postgres has information_schema views and system catalogs which you can query like regular tables.

The information_schema is actually defined by the SQL standard, so other SQL-compliant databases should have them too. I tried looking first at columns, then constraint_column_usage. Neither had quite what I was looking for, as they did not include the constraint definition.

Then I found out that each information_schema is actually just a view based upon Postgres’ system catalog tables 🤯 For example, check out the SQL for constraint_column_usage.

This meant I had the building blocks to make exactly what I needed! The pg_node_tree column contains a representation of the CHECK definition, which can be parsed with pg_get_constraintdef:

SELECT pg_get_constraintdef((
  SELECT c.oid
  FROM   pg_constraint c
  WHERE  c.conrelid = 'my_table'::regclass
));

(Note to self: the additional parentheses is to evaluate the select result as a value expression / scalar, as opposed to a table.)

That would give you a result like:

pg_get_constraintdef
`CHECK ((description <> ''::text))`

Appendix

Pro-Tips

Twitter user ascherbaum has pointed out that when using psql, you can use -e -E arguments to show the underlying SQL which Postgres uses. Pretty neat!

Follow-up Questions

Are the catalogs stored like any other table, or is there some magic that makes them table-like?
Does pg_get_constraintdef work with other types of constraints? The docs suggest not.

Don’t give up on your Rails Generators - customise them instead!

Sun, 14 Mar 2021 00:00:00 GMT

Many Rails devs I’ve spoken to quickly abandon the built-in generators. The common reason? Generated files no longer suit the project, so it’s easier to start from scratch or copy and paste. This is a pity, as generators can improve team productivity and encourage consistency, and it’s actually quite straightforward to modify them.

The Rails docs cover customising generators, but it can feel like a lot to understand if all you want is to make some small tweaks. An easy approach is to copy and override the existing generator, which I’ll walk through in this post.

TL;DR on overriding Rails Generators

Copy over the Generator classes to /lib/generators/... and modify
Copy over the template file to /lib/templates/... and modify
Bonus: Use hook_for to generate a test file if the generator doesn’t do it automatically

Background

At work, we use rake tasks quite a fair bit, both for cron-like jobs and one-off migration or support tasks. Rails has a built-in task generator, with the form:

bin/rails g task file_name [action]

It’s never quite worked for us:

We like to keep one task “action” per file, with the file named the same as the action - this makes it easier to grep for the task. The built-in generator requires separate file_name and action arguments, which is a little unnecessary.
We sometimes namespace tasks e.g. with a “onetime” namespace for one-off tasks, but the built-in generator ignores paths in the file_name and also uses the file_name as the namespace.
We write tests for our rake tasks, but the generator doesn’t create test files.

Let’s address these! We chose to override the built-in generator entirely, but you could also create a new generator instead.

Overriding the Task Generator

Rails searches for generators in a few paths, so as long as we place our file in /lib/generators/task/task_generator.rb, Rails will load our custom file instead. (Note: if you wanted to create a new generator instead, rename the files and classes according to the lookup logic above!)

To start, copy over the original task generator to use as a base. Most of the default generators can be found in railties/lib/rails/generators/rails, the one we’re after is here:

module Rails
  module Generators
    class TaskGenerator < NamedBase # :nodoc:
      argument :actions, type: :array, default: [], banner: "action action"
    
      def create_task_files
        template "task.rb", File.join("lib/tasks", "#{file_name}.rake")
      end
    
    end
  end
end

Copy it over to /lib/rails/generators/task/task_generator.rb in your project. As a simple sanity check to see if we’ve overridden the class, let’s make the generator spit out a hardcoded file name. Change this line:

template "task.rb", File.join("lib/tasks", "hardcoded_file_name.rake")

Then run the command:

bin/rails g task path/to/my_task
# create lib/tasks/hardcoded_file_name.rake

It works! Revert the hardcoding and leave the rest of the class as-is for now. We’ll be back, promise.

Overriding the Generator Template

We want a custom template as well, so let’s override that. When invoked, generators search for templates in a different set of paths from the invoker (I explored this more in another post, this works for templates from gems as well!) - we’ll be wanting to place our template at /lib/templates/rails/task/task.rb.tt.

Copy the original and modify it as such - the lack of indentation is intentional:

<% class_path.each_with_index do |path_fragment, index| -%>
<%= indent("namespace :#{path_fragment}", index * 2) %> do
<% end -%>
<% content = capture do -%>
desc "TODO"
task <%= file_name %>: :environment do
end
<% end -%>
<%= indent(content, class_path.size * 2) %>
<% (0...class_path.size).reverse_each do |index| -%>
<%= indent('end', index * 2) %>
<% end -%>

Let’s breakdown the changes. The outer tags implement our desired name-spacing, including the indentation (note that indent is actually a private method on Rails::Generators::Base, so you won’t be able to use it in your regular view templates):

<% class_path.each_with_index do |path_fragment, index| -%>
<%= indent("namespace :#{path_fragment}", index * 2) %> do
<% end -%>
...
<% (0...class_path.size).reverse_each do |index| -%>
<%= indent('end', index * 2) %>
<% end -%>

With the previous command bin/rails g task path/to/my_task, the above namespaces the task correctly:

# /lib/tasks/my_task.rake
# file is in the wrong location, but we'll sort this out later
namespace :path do
  namespace :to do
    ...
  end
end

The enclosed tags and code are for the actual task:

...
<% content = capture do -%>
desc "TODO"
task <%= file_name %>: :environment do
end
<% end -%>
<%= indent(content, class_path.size * 2) %>
...

Here, we indent the task description strings by first assigning it to a variable, content via the capture helper. Then, we indent and insert it into the file. So running the command again will give us:

namespace :path do
  namespace :to do
    desc "TODO"
    task my_task: :environment do
    end
  end
end

Our file is still being created as /lib/tasks/my_task.rake though, instead of /lib/tasks/path/to/my_task.rake.

Fixing the Generated Path

To fix the path, we make some changes to the generator:

4,5d3
< argument :actions, type: :array, default: [], banner: "action action"
<
7c5
< template "task.rb", File.join("lib/tasks", "#{file_name}.rake")
---
> template "task.rb", File.join("lib/tasks", class_path, "#{file_name}.rake")

Above, we’ve removed the unnecessary actions argument and added the class_path to the generated file’s path. Running the command again nests the file in the right folder now.

Hooking in a Test file

“Should you test rake tasks?”… is not a discussion for this article, let’s assume you do! The built-in generator doesn’t create a test file, but we can use hook_for, which lets the generator invoke other generators. Add the following DSL to your generator class:

hook_for :test_framework

This attempts to invoke Rspec::Generators::TaskGenerator, which… doesn’t exist yet. Let’s resolve that.

Adding RSpec Generator and Template

RSpec doesn’t have a custom generator for tasks, but the steps for overriding it are more or less the same. Let’s start by creating the TaskGenerator class in /lib/generators/rspec/task/task_generator.rb:

# frozen_string_literal: true
require 'generators/rspec'

module Rspec
  module Generators
    class TaskGenerator < Base # :nodoc:
      def create_task_specs
        template "task_spec.rb", File.join("spec/tasks", class_path, "#{file_name}_rake_spec.rb")
      end
    end
  end
end

Next, we need to create the corresponding template in /lib/templates/rspec/task/task_spec.rb.tt (we can definitely DRY this out more, you might be able to draw some inspiration from these examples by Thoughtbot and 10Pines):

require 'rails_helper'
require 'rake'

describe '<%= class_path.join(':') %>' do
  describe ':<%= file_name %>' do
    before(:all) do
      Rake::Task.define_task(:environment)
      Rake.application = rake = Rake::Application.new
      rake.rake_require 'tasks/<%= File.join(class_path, file_name) %>'
      @task = rake['<%= class_path.concat([file_name]).join(':') %>']
    end
    
    it 'works' do
      @task.execute
    end
  end
end

With that done, running the generator command now creates the files just as we wanted. Neat!

Go forth and generate!

I hope this has given you some inspiration to give generators a second chance - with a little tweaking, they can be really useful in saving time for you and your team. What ideas do you have for improving your generators?

How does ActiveRecord know the primary_key?

Sat, 13 Feb 2021 00:00:00 GMT

While trying to help someone debug an ActiveRecord::UnknownPrimaryKey error, I went down a rabbit-hole trying to figure out how Rails determines the primary key. Hopefully this provides a tiny bit more insight on how Rails works!

TL;DR

If you’re using Postgres, ActiveRecord searches for a primary key index on the table. If you’re using a different database, the answer probably lies in the respective SchemaStatements of the adapter, e.g. ActiveRecord::ConnectionAdapters::MySQL::SchemaStatements (source).

Finding the Primary Key

ActiveRecord::Base includes the AttributeMethods module, which in turn includes PrimaryKey. The module has a #primary_key method:

def primary_key
  @primary_key = reset_primary_key unless defined? @primary_key
  @primary_key
end

If we follow the #reset_primary_key method, we’ll eventually reach #get_primary_key, which tries to get the key from the SchemaCache:

def get_primary_key(base_name)
  if # ...
  else
    if ActiveRecord::Base != self && table_exists?
      pk = connection.schema_cache.primary_keys(table_name)
			# ...
    else
      # ...
    end
  end
end

In the SchemaCache, we see that if the primary_keys cache does not already have the key, it will attempt to call connection.primary_key(table_name):

def primary_keys(table_name)
  @primary_keys.fetch(table_name) do # if the table_name key isn't in the Hash, execute this block
		# ...
    @primary_keys[deep_deduplicate(table_name)] = deep_deduplicate(connection.primary_key(table_name))
    # ...
  end
end

This method is found in SchemaStatements, and it in turn attempts to call #primary_keys. This call goes to the database adapter, in my case PostgreSQL::SchemaStatements:

def primary_keys(table_name) # :nodoc:
  query_values(<<~SQL, "SCHEMA")
    SELECT a.attname
      FROM (
             SELECT
               FROM pg_index
              WHERE indrelid = #{quote(quote_table_name(table_name))}::regclass
                AND indisprimary
           ) i
      JOIN pg_attribute a
        ON a.attrelid = i.indrelid
       AND a.attnum = i.indkey[i.idx]
     ORDER BY i.idx
  SQL
end

The subquery in particular is crucial - basically, Rails looks for a primary key index belonging to the table. Let’s focus on the important bits:

SELECT ...
/* from all the indexes */
FROM pg_index
/* find those for table_name */
/* you can google for regclass, it's another Postgres rabbit-hole */
WHERE indrelid = table_name::regclass
/* that are primary keys */
AND indisprimary

This is usually automatically created during a migration - Rails will add an id column and make it a primary key. If you have a structure.sql file, you’d find statements like ADD CONSTRAINT table_name_pkey PRIMARY KEY (id) which do exactly that.

Bonus: How did I find it?

It took me quite a while to figure this out, I’m sharing my abridged (because I was flailing around a lot) methodology in hopes that it might be helpful to you when debugging - if you know a better way, do share!

Did a GitHub search in the Rails repo for the UnknownPrimaryKey error. There were a few results that were in the form finder_class.primary_key, so I assumed it was a method on the model (and confirmed that by playing around in the rails console).
Did another search for "def primary_key" (“primary_key” alone gave too many results), this led to ActiveRecord::AttributeMethods::PrimaryKey.
Wasn’t sure that this was the correct method but I attempted to follow the code through (GitHub’s jump to definition was a godsend) and got a little stuck/distracted by trying to figure out what class connection referred to.

After a bit of poking around in the console, I recalled seeing this class called TracePoint, and decided to give it a try:

trace = TracePoint.new(:raise) do |tp|
    p [tp.lineno, tp.event, tp.raised_exception]
end

trace.enable { ExampleModel.first }

This gave an extremely long trace 🤯 good thing I could search in my terminal, after grepping for primary_keys I came across this set of calls, which confirmed that the method was in PostgreSQL::SchemaStatements

[48, ActiveRecord::ConnectionAdapters::SchemaCache, :primary_keys, :call]
[53, ActiveRecord::ConnectionAdapters::SchemaCache, :data_source_exists?, :call]
[146, ActiveRecord::ConnectionAdapters::SchemaStatements, :primary_key, :call]
[355, ActiveRecord::ConnectionAdapters::PostgreSQL::SchemaStatements, :primary_keys, :call]

Window Functions, Visualised - Rankings

Mon, 04 Jan 2021 00:00:00 GMT

Leading from an introduction to Window Function Calls, let’s dive deeper into the different Window Functions that are available. Today, we’ll look at row_number, rank, dense_rank and percent_rank. We’ll be using the same expenses table from the previous post, which had entries like:

description	cost
bus ride	3
lunch	15
…

In the following examples, our window frame will be the whole expenses table, ordered by highest to lowest cost:

SELECT
  [ window function ] OVER(ORDER BY cost DESC),
  expenses.description,
  expenses.cost
FROM expenses

row_number

Let’s start with row_number:

... ROW_NUMBER() OVER(ORDER BY cost DESC), ...

All row_number does is add a running number for each row in the frame, starting from 1:

row_number	description	cost
1	groceries	60
2	dinner	35
3	taxi	20
4	lunch	15
5	lunch	15
6	supper	15
7	tea break	5
8	bus ride	4
9	bus ride	3
10	bus ride	3

dense_rank

dense_rank is more interesting. Reviewing the previous example, we can see that there were a few entries that have the same cost. The Postgres documentation refers to these as peer groups, and dense_rank adds a running number counting by the groups instead of rows:

So changing the SQL to this:

... DENSE_RANK() OVER(ORDER BY cost DESC), ...

Gives us this:

dense_rank	description	cost
1	groceries	60
2	dinner	35
3	taxi	20
4	lunch	15
4	lunch	15
4	supper	15
5	tea break	5
6	bus ride	4
7	bus ride	3
7	bus ride	3

Note: I’m actually not sure how Postgres orders the rows within a peer group - in my small example, it looks like an implicit ORDER BY ID DESC was added, but the Postgres docs also say for general ordering, if sorting is not chosen, the rows will be returned in an unspecified order 🤷‍♂️.

rank

rank has one big difference from dense_rank - it counts the “gaps” in the previous peer group:

Let’s compare the results of each function to show the difference:

dense_rank	rank	description	cost
1	1	…	…
2	2	…	…
3	3	…	…
4	4	…	…
4	4	…	…
4	4	…	…
5	7	…	…
6	8	…	…
7	9	…	…
7	9	…	…

percent_rank

This is an interesting one - how it’s derived is a bit of a mouthful, so let’s show by example, starting with the query:

... PERCENT_RANK() OVER(ORDER BY cost DESC), ...

And the result:

percent_rank	description	cost
0	groceries	60
0.11…	dinner	35
0.22…	taxi	20
0.33…	lunch	15
0.33…	lunch	15
0.33…	supper	15
0.66…	tea break	5
0.77…	bus ride	4
0.88…	bus ride	3
0.88…	bus ride	3

Instead of a running number, we get the “relative rank” from 0 to 1 inclusive. This is useful if you need to know how close something is to the highest rank - though, as the above example shows, it may not end with 1 if there are ties for the highest rank (a deeper explanation here).

side-by-side

Finally, let’s look at results side by side:

row_number	dense_rank	rank	percent_rank	description	cost
1	1	1	0	groceries	60
2	2	2	0.11…	dinner	35
3	3	3	0.22…	taxi	20
4	4	4	0.33…	lunch	15
5	4	4	0.33…	lunch	15
6	4	4	0.33…	supper	15
7	5	7	0.66…	tea break	5
8	6	8	0.77…	bus ride	4
9	7	9	0.88…	bus ride	3
10	7	9	0.88…	bus ride	3

I hope that helped! Here’s an sqlfiddle you can play around with.

In future articles, we’ll look into more window functions. Follow me on Twitter to be informed of the next one!

Window Function Calls in Postgres - A Visual Introduction

Thu, 31 Dec 2020 21:56:03 GMT

What is a window function call? My explanation:

adds a column to query results that is calculated based on other rows in the table

Ok, that’s probably still not immediately understandable, so let’s walk through an example. Imagine you’ve been tracking expenses in Postgres:

description	category	created_at	cost
bus ride	transport	2020-01-01	3
lunch	food & drinks	2020-01-01	15
…

Now, say you wanted to see how much each expense compares to the average:

description	category	created_at	cost	avg
bus ride	…	…	3	17.5
lunch	…	…	15	17.5
…

You could use a subquery to calculate the average:

select *, (select avg(cost) from expenses)
from expenses;

But we could also be a little fancier and use a window function call:

select *, avg(cost) over()
from expenses;

Although we have some new syntax to understand (which we’ll get into later), that’s noticeably shorter! What just happened?

Hello, Window Frames

Let’s try to visualise the window function call. For each row, Postgres calculates the average cost of the rest of the table, which is the default “window frame”:

Now if that was all you could do with window function calls, I’d be disappointed too. Thankfully, it’s not! Aside from being shorter, there are some key features that are harder to replicate with subqueries:

calculations can be done over a subset of the table (i.e. a smaller window frame) instead of the whole table
in addition to the usual aggregation functions like avg or max, there are window functions, which can only be used as part of a window function call

Reframing the Window

Let’s try calculating over a subset. Knowing the average across all expenses wasn’t very helpful - a big purchase could skew the calculation, so let’s instead compare each expense to the category average:

select *, avg(cost) over(partition by category) as "avg in category"
from expenses;

Our result now looks something like:

description	category	created_at	cost	avg in category
tea break	food & drinks	…	5	24
lunch	food & drinks	…	15	24
…
taxi to home	transport	…	20	7.5
bus ride home	transport	…	3	7.5
…

That’s a bit more helpful! Let’s visualise this:

For each row, our new window frame includes only other rows with the same category, and the average is calculated based on those rows.

This is also a good time to take a brief look at the syntax we’ve used so far:

..., avg(cost) OVER(PARTITION BY category), ...

In the above:

avg(cost) is the aggregation function that we want to execute over the window frame - we could use other functions such as max or min
OVER is the keyword which tells Postgres we were using a window function call. Everything in parentheses specifies how our table will be divided into different window frames
in this case, we are using PARTITION BY category, which means we are dividing the table into different window frames based on their category

Window Functions

Now that we know a little more about the syntax, let’s look at the other hidden power of window functions calls… window functions themselves!

It’s coming to the end of the month, and you want to know the biggest expense in each category, for each day. There’s a window function for that - rank!

With rank, we can calculate the order of each row in it’s window frame (Note that the following query still returns every row in the table, not just the top expenses - we’ll fix that later):

select
  *,
  rank() over(
    partition by category, created_at
    order by cost desc
  )
from expenses;

Notice the additional syntax in our window function call:

... OVER(PARTITION BY category, created_at ORDER BY cost DESC) ...

we’ve added a second condition to PARTITION BY to further divide each window frame by the day each entry was created_at (this is very similar to GROUP BY)
there’s now an ORDER BY ... DESC that orders the rows within each frame from highest to lowest cost, so we’ll know that a rank of 1 would be the most expensive

Another quick visual:

So for each row, our window frame is other rows with the same category and created_at date, and we order the rows from highest to lowest cost.

The query gives us a result like:

description	category	created_at	cost	rank
groceries	food & drinks	2020-01-01	60	1
dinner	food & drinks	2020-01-01	35	2
lunch	food & drinks	2020-01-01	15	3
…
taxi	transport	2020-01-01	20	1
bus ride	transport	2020-01-01	3	2
bus ride	transport	2020-01-02	4	1
…

Now that we have a ranking, we could use it as a subquery and filter out expenses that are rank 2 or higher.

select *
from (
  -- our earlier query
  ...
) as "ranked_expenses"
where rank = 1;

Which leaves us with exactly what we want - nice!

description	category	created_at	cost	rank
groceries	food & drinks	2020-01-01	60	1
lunch	food & drinks	2020-01-02	15	1
taxi to home	transport	2020-01-01	20	1
bus ride home	transport	2020-01-02	4	1
…

I hoped that helped you grok window functions! If you want to play around with the data in this example, there’s an sqlfiddle - in case that doesn’t exist at the time of reading, you can also get the SQL to create the table and query the data from this gist.

In the next few articles, we’ll look into the individual window functions, starting with those for ranking rows.

2020 in Retrospect: Making

Fri, 11 Dec 2020 20:48:03 GMT

Reflections on side projects this year.

Overall

I tend to build first (out of technical interest or my own personal problem), then make only half-hearted efforts to validate and abandon them quickly. There’s two takeaways for me: I should make a proper effort to get better and more comfortable with customer discovery and idea validation, and if it’s to validate an actual business I should do that before building. Will be trying out some suggestions around that.
The “technical” projects were great for personal learning, and fun (until they got serious). I still have a lot to learn and catch up in dev, so I think it’s worth setting aside time for these. Maybe a dual-track thing - set aside 50% time to validate ideas, 50% time to improve technically? Or maybe I should try to capitalise on the learnings e.g. teaching others?
I abandoned some projects once it became clear they took more effort than expected, so I didn’t see myself making money from them soon. Perhaps in addition to the “dual-track” approach, the ideas I should try to validate first should be more around “simpler, one-time sale” products as Rob Walling suggests. That might take off some pressure on pure-tech projects so I can go deeper.
I get quite discouraged and annoyed with design. I am trying to get better at it for my Dec project, but not in a particularly structured manner. I do think it’ll be useful, just not sure if I should be prioritising that over other things.

Mar 2020

A “self-hosted” Kahoot! clone

In March, I helped out at a church event where we found out Kahoot had unexpectedly restricted the free plan. I thought it’d be cool if there could be a “self-hosted” version for non-profits, and made a really non-user-friendly-but-free prototype with GitHub Pages / React / Airtable
It had some real UX deficiencies to make it free for me to host, so I wanted to have some testers to see if the tradeoffs were ok before deciding to promote it
I asked a friend to try it but didn’t really follow-up, and also made a post on Reddit (r/react), more a technical post (not much interaction there though)
Reflections:
- it was fun to give the Airtable API a try, also a good opportunity to get a little more familiar with Promises
- I definitely didn’t feel comfortable promoting the initial version of the project, there were so many hoops someone had to jump through just to even try it (setting up a throwaway Airtable account for example). In hindsight, this was probably a mistake - these are all guesses >_<

Apr 2020

There were some really useful newsletters I had been following, but I wished there was a way to consume the articles shared in a more piecemeal manner. For example, Hackernewsletter includes >50 articles each week, I usually end up opening 15 tabs in my mobile Firefox and then slowly consuming them over the week - which you can guess isn’t very successful.
Spent a week or two repurposing an old project and started scraping the web archives of Hackernewsletter. (I later found out that Hackernewsletter had stopped publishing their archives, so that wouldn’t really work)
Showed the idea to a friend, who gave some really useful feedback (being able to retrieve the knowledge when it would be useful was a bigger problem than consuming it)
Also, I couldn’t think of a way to give due credit back to the newsletter creators, unless I worked directly with them (feels like a chicken and egg thing) and I didn’t want to disrupt their income source.
Reflections:
- The turnaround time was pretty OK (2 weeks) so not a huge loss
- However, I probably should have led by talking to more people instead of building the backend first - since it was mostly repurposing I didn’t really learn anything new from the technical standpoint
- Also, I didn’t really pursue the knowledge surfacing problem that he raised, because it was pretty nebulous and I was… lazy. Looking at the emphasis on knowledge management solutions now, it does seem like that was another mistake 😞

Jun 2020

A “self-hosted” real-time Kahoot! clone

This actually still works, feel free to give it a try!

I thought it would be an interesting technical exercise to try and make the previous project real-time. Wanted to try something more DIY like socket.io, ended up using Firebase.
I didn’t really do much validation except showing it to 2 friends, with a lukewarm response
I also decided against pursuing it further in it’s current form:
- Competing with Kahoot! on price was probably not going to work, their plans were still pretty affordable (and it seems, even cheaper now)
- They were focusing on education, so I thought another angle was a casual party game, which incidentally was how I was introduced to Kahoot. Something like an MMO bishi-bashi. I shelved this for a future date because I felt like I didn’t have the expertise for building web games, so it would be a long-build.
Reflections
- Even if it wasn’t exactly what I initially wanted to learn (Firebase abstracted a lot of the real-time aspects), I did walk away with some more knowledge of NoSQL databases, plus some Firebase-specific knowledge (e.g. security rules), which was great.
- Again, didn’t really push on the validation front, possibly because I felt it needed more polish, plus I wasn’t sure if I was going to stick with Firebase if moving to the variety-game idea
- I do feel like the rise of games like Fall Guys and Among Us somewhat validates the MMO casual game… 🤔

Jul 2020

Over the Kit

This is still alive! https://wasabigeek.github.io/drumz/

Me being a very lazy drummer, just want to be given some licks to practice instead of having to orchestrate and build patterns up to speed myself
Built a small, limited prototype (only 4/4, only 16th notes 1 bar fill) then tried to do some validation
- Posted on Reddit r/drums, relatively speaking there was very little interest (few upvotes and comments compared to what seems to usually trend there)
- Pinged 3 musician friends - feedback was more in the “oh this is cool” sorta vein (I need to dig up those messages)
Made some upgrades (the biggest being multiple usages: part-bar, 1-bar, 1-bar 16th note triplets variations)
Reflections:
- I used Vexflow and WebAudio, which was really interesting
- r/drums might not be the right subreddit, given the focus there seems to be showing off 😋 maybe I should try r/drumming (which has less people but seems more educational) or see if there are some groups / chats I can join to talk with people

Aug 2020

HabitStar

I never made it to releasing on the App Store, but some screens of the prototype still exist on Makerlog

The thought process was something like, I’ve always wanted to try to build a mobile app + mobile apps seem to make a nice indie hacker project (could push infra to Apple via saving on device/iCloud, so it sounded like it’d be easier for one person to maintain than a web app)
Initially thought to convert Over the Kit to an app, but after working with and digging into React Native for a bit, it seemed like it would be a challenge (due to Javascript’s event loop), so I thought I’d start with something simpler
I always wanted to make a habit tracker, because the one I was using still felt a little clunky
Got a little bogged down trying to (i.) make a design I liked; (ii.) figure out the gamification mechanics (which I wanted to have in order to differentiate it). My lack of experience around mobile development and React Native (data storage, testing and debugging) made the experience slow and frustrating at times as well.
At one point I was wondering whether to set the gamification part aside for a bit to just focus on getting the app out. But I thought the gamification was part of the charm, and didn’t want to release without it (the $99 developer fee also was a small discourager)
In the end, I just drifted to something else…
Reflections:
- The design part especially seemed to bog me down mentally (“I wish it looked better”). Maybe I shouldn’t let myself get too discouraged with a sub-standard design. Or maybe I should try to get more comfortable with design?
- I’m sure the “slow and frustrating” feeling contributed to me drifting away - but I should actually try to get used to being in that zone more, because it probably means I’m doing something way out of my comfort zone and learning something new.
- I finally got to try React Native after thinking about it for years, so that was great! I do wonder if the tooling around Flutter is better though, maybe will give that a bit of a try. If it leans towards the game aspects more too, I wonder if I should try something like Unity 🤔 All of the above sound like pretty significant time commitments.

Aug/Oct 2020

ClassCal

Still a work in progress, there’s a staging app you can try, plus a quick and dirty wix landing page my friend made https://www.classcal.com/

Discussed a few ideas with a friend before we decided to give a Tuition scheduling app a try, based on some earlier conversations he had while validating a different idea. For me, I was also curious about whether we could work together, since he had some experiences starting more traditional businesses, and seemed to be stronger in the sales aspect.
In terms of validating the idea:
- my friend did some demos with existing solutions in the market - we thought a possible angle was to try to do something more affordable and SaaS-like, since the incumbent solutions in the market were positioned more as custom solutions, with prices to match (though they could qualify for the govt. grants)
- we spoke with 2 of his contacts in the industry, the second one was positive but also seemed to have a non-traditional approach to the business, managing it more like a gym with online bookings and payments
- we tried to walk the ground and speak with some tuition centres, but this was super hard. Most people just shoo-ed us away (to be fair, it was true that they were busy)
- my friend also tried to post a poll in a Facebook groups, there was little response
- now we’re trying Google Ads with the landing page in a last-ditch attempt to get people to talk to and learn from, before we decide whether to move on to a different idea
We agreed not to build anything beyond a Figma prototype until we had more validation, but I had itchy fingers by October and went to start on the app. It was a nice chance for me to figure out how to build a SaaS in Rails as well (I had tried some out-of-the-box template with the newsletter idea but it was not well-documented and I didn’t know enough / need all of the bells and whistles)
Reflections:
- I think it’s worth spending time aligning with a partner on approaches and beliefs. Personally, I realised I didn’t really know how to explain processes I had taken for granted, such as user story mapping.
- It was helpful having the second brain, we weren’t always on the same wavelength but it did push me to have a different perspective and read up stuff I’ve been putting off e.g. on validation
- I think I will still continue building for a bit, because it does provide a potential future framework for launching other projects. In the worst case, I will open source it and try to do some walkthrough to building a SaaS in Rails.

Sep 2020

PeekPalette

The unfinished landing page is still up

While doing the Figma prototype, I spent quite a bit of time figuring out colour variations and thought, there must be an opportunity here - an app that would generate and let you preview colour combinations against a UI (something like Headlime, but for UI colour palettes)
I decided to try starting from a landing page and validate that first. After spending a week on an interactive hero section, I realised I didn’t know what else to put in the “marketing” section of the site that would be convincing >_<
In terms of validation, I did a half-hearted post on Makerlog and another on IndieHackers, again, very little feedback or interest.
Also thought that the target might be so niche it didn’t exist: If I’m a dev that wasn’t good with design, I’d probably purchase some theme instead.
I do think there’s some opportunity in this market, but this probably wouldn’t cut it as a standalone product
Reflections:
- I did enjoy working on the hero section, got to try out a rainbow CSS gradient I saw on another blog
- I didn’t enjoy trying to figure out the design of the UI on the interactive element, and also figuring out the marketing copy. That’s also probably the point when I started to realise the idea didn’t have legs. In this case, might have been helpful to validate more first before building, but I think I did ok here, didn’t “waste” too much time.

Dec 2020

BaseUI (working title)

Continuing the theme of “I wish there was an easier way to do a nice design”, I’m currently trying to validate whether my problem of getting a decent but still unique design done quickly applies to other IH devs:
- Inspired by the Mom Test, I tried not to rush into building, but tried to understand the target’s (IH devs) current workflow a bit more with a post on IH https://www.indiehackers.com/post/serial-saas-devs-whats-your-workflow-when-starting-a-new-project-7d4c43a2a3 (it was also useful for me)
- I looked at other solutions in the market: full-app templates like JumpstartPro and pure UI component libraries like TailwindUI, thought there might be an opportunity to do something in the middle.
- I’m now trying to build (oops) a free, minimal set of components that’s integrated with Rails, and seeing whether there’s interest in that before going further. Again, struggling a little with design here (I would love to just plonk TailwindUI in there, but it goes against their license, and I’m not sure if there’s a cheap way to get a neat set of components)

Customising Generator Templates in Rails (Gems too!)

Sat, 07 Nov 2020 20:32:03 GMT

Rails’ generators are a great boon to the Indie Hacker^TM or Solo Developer^TM. Running rails g scaffold ... will quickly bootstrap the full stack for a model, creating the routes, controllers, views and even tests.

As you start customising and standardising your interface though, the default templates get less and less useful - most of the generated code doesn’t fit your layout or setup, and you end up deleting most of it or skipping them.

What if we could change the… templates?

Rails actually gives you a lot of control over the generators, but early on in a project, perhaps the most impactful and simple thing to do is to override the templates.

Let’s look at an example. Normally Rails would generate a show.html that looks something like this:

<p id="notice"><%= notice %></p>

<p>
  <strong>Attribute:</strong>
  <%= @object.attribute %>
</p>

<%= link_to 'Edit', edit_object_path(@object) %> |
<%= link_to 'Back', object_path %>

For a personal project, I’ve made mine generate something like:

<% content_for :breadcrumbs do %>
  <div class="...">
    Home > <%= @object.attribute %>
  </div>
<% end %>

<% content_for :heading do %>
  <div class="...">
    <h1 class="...">
      <%= @object.name %>
    </h1>
    <div>
      <%= link_to 'Edit object', edit_object_path(@object), { class: '...' } %>
    </div>
  </div>
<% end %>

<% content_for :main_content do %>
  <div class="...">
    <%= @object.attribute %>
  </div>
<% end %>

It’s saved me a lot of time!

To do so, simply add a template to your Rails project in lib/templates/erb/scaffold/show.html.erb. You can use the original as a starting point. The other templates can also be easily changed following a similar path. Note the double percentages (%%) used to escape ERB tags!

Bonus: Gems too!

This also works for your gems that have generators (e.g. rspec-rails or react-rails), though it’s not obvious where to place the templates.

Our answer lies in Rails::Generators::Base.inherited:

def self.inherited(base) #:nodoc:
  ...
  Rails::Generators.templates_path.each do |path|
    if base.name.include?("::")
      base.source_paths << File.join(path, base.base_name, base.generator_name)
    else
      base.source_paths << File.join(path, base.generator_name)
    end
  end
  ...
end

So, in addition to the source_root that most Generators would define, Rails will also search a few extra paths. In general, you should place the template in lib/templates/<top_level_module>/<generator_class_prefix>/<template_name>.

For example, react-rails has React::Generators::ComponentGenerator, so if I wanted to replace component.es6.jsx, I would place it in lib/templates/react/component/component.es6.jsx.

I hope that helps!

First Steps into Open Source

Wed, 06 May 2020 22:12:03 GMT

At start of the year, I set a rather nebulous goal:

“become a regular open source contributor”

The motivation was to experience different codebases to gain more experience and (hopefully) improve my employability. 5 months on, I feel reasonably comfortable stating that I’m an open source contributor, and hope to continue doing so.

This is a summary of my journey so far, I hope it’ll be useful for you, especially if you’re thinking of contributing!

First Timers Only

To start, I remembered reading about a first timers “movement” and quickly found a compilation of participating projects. Ruby was what we used at work, so I jumped to that section.

The first PR I tried to submit was for Rails (seemed like it would be something nice on the resume). Rails uses the label good first issue, so I picked an issue near the top, spent a Saturday working and submitted it!

Then I waited.

And waited.

Good things…

In hindsight, that issue was not a great first pick. It already had some attempts, with the most recent just a week before my PR. I think I was so unsure of myself, I figured this would be a super low-risk issue to try. After all, if I screwed it up, there were other attempts that could be picked up.

In the end, one of the maintainers took it upon himself to close the issue and credited all the contributors. His PR was better, but I guess it was mainly a way of navigating this long-drawn issue without stepping on any toes.

While waiting, I decided to try again. We were using a tool written in Go to manage our AWS IAM at work, but there was a good-to-have feature we wanted and an issue for it. So I spent a few week nights and weekends learning Go, doing up the PR and submitting it.

Then I waited.

…come to those…

And someone reviewed! I was energised. I made some changes.

Then he requested a second opinion from another maintainer.

It’s still there, open.

At this point I was starting to get disheartened. (No hate to open source maintainers, it’s pretty thankless work and the world is better for your contributions.)

…who wait keep trying.

I flapped around a bit, and one day decided to look at another gem our company had been using. I saw a pretty steady stream of merges, neatly labelled issues, plus it just felt less intimidating than Rails.

A few days later, my first merge! I was elated.

Huge props to dblock, who somehow manages to turnaround and reply quickly, often within a day or two. Rubyists, if you would love to contribute but don’t know where to, might I humbly suggest anyone of dblock’s projects - it’s a great place to start.

Closing Thoughts

For “first timers” looking to contribute, my takeaways (sample size of 1, take with a pinch of salt):

Labels on recent issues seemed like a good sign that a project is well-maintained. Search for a first-timers-only type label - for example, in Rails it was good first issue.
Check the CONTRIBUTING.md documentation first before starting on your PR! There’s usually a lot of helpful setup stuff there. For example, Rails was pretty amazing for this, I was kinda worried going in but it was really smooth, they even had instructions on how to use Vagrant to get started.
Expect a little trial-and-error to getting your first PR merged, but stick with it! I found the experience super rewarding, and indeed got exposure to different codebases and languages.
Make life as easy for maintainers as possible - try to figure out as much as you can first, whether it’s looking at previous PRs, making GitHub checks pass.

I hope that was helpful! Feel free to drop a note on Twitter to discuss / disagree :)

dense_rank	rank	description	cost
1	1	…	…
2	2	…	…
3	3	…	…
4	4	…	…
4	4	…	…
4	4	…	…
5	7	…	…
6	8	…	…
7	9	…	…
7	9	…	…

dense_rank	rank	description	cost
1	1	…	…
2	2	…	…
3	3	…	…
4	4	…	…
4	4	…	…
4	4	…	…
5	7	…	…
6	8	…	…
7	9	…	…
7	9	…	…

wasabigeek's blog

Using the XReal AIR 2 Pros for Remote Work (Programming)

The Setup

Using the XREALs unplugged

Using the XREALs without the Beam

Using the XREALs with the Beam

Other Notes

Summary

How I'm dealing with Self-Doubt as a Software Developer

Remember we don’t see the journey they took to get there

Realising they made some learning tradeoffs

Getting more feedback

Reframing as something that can be improved (Growth Mindset)

Visitor Pattern in Ruby: Examples from Rubocop

What is the Visitor design pattern?

What is Rubocop?

Object Structure: Abstract Syntax Tree (AST)

Visitor: Cops

Execution & Traversal

Tradeoffs

Benefit: Encapsulation of each Linting Rule in a Visitor

Benefit: Ease of extension by adding new Visitors

Cost: Coupling to Object Structure

Cost: Reduced encapsulation of Object Structure

Afterword

Overspecifying Tests - the Employee Report Kata

What is Overspecifying?

The Kata

First Requirement: 18 and above

Second Requirement: Sorted Names

Third Requirement: All Caps

Optional: Refactoring to Objects

Fourth Requirement: Sort Descending

Takeaways

Afterword

Coupling is a Tradeoff

What is Coupling?

An Example: Extracting Twitter’s “Algorithm”

Not all coupling is discernible from the code

Coupling is difficult to completely eliminate from a large system

Refactoring is important

Afterword

Zettelkasten from "How to take Smart Notes"

Key Takeaway: Purposeful Note-taking

Summarising Zettelkasten

1a. Take Fleeting Notes

1b. Take Literature Notes

2. Write Permanent Notes

3. Connect and Index Permanent Notes

4. Develop ideas in Permanent Notes

5. Write from your ideas

The Review

Why use Polymorphism in OOP?

What is polymorphism?

Why use polymorphism?

Afterword

A 1972 paper and the Single Responsibility Principle

The System: KWAC Index

Modularization 1: “Flowchart”

Modularization 2: “Information Hiding”

Comparison: Changeability

Conclusion

Further Reading:

Reading a Ruby gem with VSCode

Open the Source Code

Find your Target

Trace the Callers

Know the Structure of a Gem

Footnotes

Practical Metaprogramming in Ruby: minitest/mock

What is metaprogramming?

First example: Set expectations with method_missing

Remove inherited methods via undef_method

Redefining Methods with define_method

Stubbing by redefining methods on “Metaclasses”

Footnotes

Mocking in RSpec VS Minitest: A Cheatsheet

Footnotes

My Learning Workflow for 2022

1. The Workflow

dense_rank	rank	description	cost
1	1	…	…
2	2	…	…
3	3	…	…
4	4	…	…
4	4	…	…
4	4	…	…
5	7	…	…
6	8	…	…
7	9	…	…
7	9	…	…