Weekly Journal 6 - Knowledge base, Terraform, Elasticsearch

Synopsis

This week, I think I have had an epiphany with my personal knowledge base. I spent some time building some automation to create and manage AWS accounts. I am ready to retire some Elasticsearch clusters, but shutting them down is harder than it looks.

Knowledge Base

I have been struggling a bit with both the tooling and figuring out how to organize my research notes. In the past I have used Microsoft OneNote to store and manage my notes. I really like OneNote, but as my knowledge base grows, I do not want to get locked into a proprietary platform. I have put together a decent system using git, Markdown, and the Foam extension for Visual Studio Code. While I have not figured out yet how to replicate all of OneNote’s features, it is working well for managing the Markdown text files I create.

I have also been struggling with managing my research notes. I started out by organizing my notes by the source, but that has quickly turned into a bit of a mess. Instead, I am going to start organizing everything by topic. Starting with a broad topic for new items, and gradually breaking them down into finer-grained sub-topics as I add more information. It is similar to the atomic note concept from the Zettelkasten method. I am interested to see how this works out.

Terraform for AWS Account Management

I spent some time on a project I started last year to help manage our AWS accounts and organizations. Last year I setup a Terraform project and imported the current state of our AWS accounts. This week I was able to add to that project to provision new accounts and setup the initial IAM roles. There is a chicken or egg problem that I need to sort out with respect to the creation of roles in an account, but I think that can be solved by creating a separate definition for account resources after the account itself has been created.

Elasticsearch

For the past couple of years we have been using AWS’ Elasticsearch Service to store aggregated logs. I think I have regretted the decision to use Elasticsearch from the start. This has been further reinforced by the recent fiasco around Elasticsearch licensing future versions using the SSPL, a fauxpen source license and the subsequent announcement by Amazon of a project fork.

On one hand I do think Amazon should be supporting the open source projects they leverage as part of the cloud service offerings. It is really terrible that they take from these projects without giving back, either in contributions or financial support. However, using the open source community to drive adoption, forcing contributors to sign a CLA, and then leveraging that to relicense under a non-free license is pretty underhanded. For me, the takeaway is to never trust an open core project like Elasticsearch used to be. Whether it is Elasticsearch, MongoDB, GitLab, or any other “open” source project with proprietary extensions, my advice is to avoid it completely if at all possible. I would rather use a fully proprietary program or service over an open core project. At least they are being honest about their intentions.

To add insult to injury, Amazon does not have feature in the console to take snapshots of a hosted Elasticsearch database. So now I have to either write a script or fool around with Postman or something similar so I can create a final backup before we shut it down. I will be so happy once we have this removed from our stack.

What’s Next?

I think the primary focus this coming week will be getting the Elasticsearch clusters backed up and terminated. I am also looking into dumping my use of Google services. First thing on that list is to find a hosted service for my personal email.