
A few weeks ago, we started working on a project that we now want to share with you. After some deep analysis, the development team realized that the Git repository organization we had, which consisted of multiple repositories, was adding complexity to many parts of our daily work.

We previously had four projects with independent life cycles. We had the backend, frontend, mobile, and e2e tests repositories. Whenever we wanted to ship a new feature to production, the developer was responsible for deploying each subproject properly and managing the required orchestration to ensure that the application worked as expected. One example is the classic deployment of the backend, minutes before the frontend or mobile. This caused some concurrency problems when the rollout of the latest version of the backend was not ready, and the frontend app was already in production.

For this reason, we started working on the challenge to merge four of our main repositories into a single mono repository. This would bring to the development team, a better, and more cohesive way of seeing the Factorial product. It would help us to ship features on all levels of the stack with a single pull request, and much better dependency management between projects, and afford us a way to know if we would break an API after a static analysis. In general, it would bring great value to the final quality of the project.

Due to the fact that our project has a solid set of technologies in many places, it is great to share common configuration files to set in place a group of rules and conventions that help us keep raising our code quality every day.

## Embarking on the second big migration - The Dos and Don’ts

As we did with other big migrations, like the [Typescript migration](https://labs.factorialhr.com/posts/how-we-migrated-400k-lines-of-code-from-flow-to-typescript), we listed a set of *dos* and *don’ts* for this migration. Both migrations share many similarities in terms of the topics to take into account. We needed to understand that we were working with more than 100 developers and if we disrupt their daily work, it would have an unacceptable cost.

For this reason, we listed these main topics:

- **Freezing the code was not acceptable:** We can not stop the development *machine in order* to perform an action that takes an undetermined amount of time. Due to this, this migration needed to be as short as possible. As a result, we carried out all the required changes during a single weekend. All developers had a fully working repository next Monday 😊
- **Git history information had to be preserved:** The history behind each one of the commits of the four repositories has an invaluable price. Understanding the context of a commit helps developers every day. For this reason, we could not afford to lose it during the migration.
- **All open pull requests will be migrated to the new repository:** Related to the two previous points, the migration had to migrate the open pull requests. This is a challenging point when you are rewriting the whole Git history, but we made it. 💪
- **Communicate, document, and share:** One important step of this migration was to communicate this change properly to the rest of the team. Everybody needed to be acknowledged in all aspects of it. Many issues during these kinds of migrations arise from miscommunication. Therefore, it’s imperative to take the time to document and communicate all the details about it.

## How did we do it?

One approach we used in the past that helped a lot was to perform the migration by a single written down script with all the directives to consider during the migration.

- In this case, this migration script collected all the git historical information and moved it into subfolders, thanks to [git-filter-repo](https://github.com/newren/git-filter-repo). This tool helped a lot in the performance of this code migration in a reasonable amount of time. Other options like [filter-branch](https://git-scm.com/docs/git-filter-branch) were awfully slow and were discarded.

- To migrate all the existing open pull requests for each repository, we decided to interact with the GitHub GraphQL API and move all the required information into the new repository.

- To make all the required tests for this migration, we “played” in a test repository where we kept running this script and resetting all the stuff when we detected that we were not comfortable with some final results. We kept iterating with this approach until the final result was the desired one.

If you are interested in the details, we prepared a [gist](https://gist.github.com/fcsonline/6f04319b1d7b99a09eb842f7e14aa063) with all the scripts to perform this migration.

In addition, we performed many other manual actions to complete the migration. We tried to reduce the number of manual steps to as minimum as possible. If a step can be written down in the script, it is one less thing to worry about during the migration. Thinking really hard about all the things to do before the migration, no matter how small they are, was very important, and in our case, the list had almost 120 items! 😱

We need to be aware that there is always a small probability that something was not taken into account during the planning phase. In our case, we made a mistake with some of the GitHub actions workflows, where some of the working directories were not properly set.

## Conclusion

Every time you do this kind of migration, there are some thoughts that tell you not to do it 😅. As engineers, the sentence “*Don’t touch it if it works*” takes a relevant meaning during these migrations. But we overcame our fears because we played in a scenario where we were seeing the final result after each script test iteration, like a dry-run. Also, the fact that we had the backup plan of not using the monorepo at all was a great path to abort the migration.

Weeks later, it is safe to conclude that the migration was a success, and every day we get the mentioned benefits. Almost all developers were not affected by this migration and all our environments were ready to work the next Monday.


A short explanation how we merged our main repositories into a single monorepo - the reasons, considerations, dos and don’ts, challenges, and wins.


How we merged our main repositories into a single monorepo


When Factorial was born, there was a *battle* in the web ecosystem to decide which JavaScript type system was the *de* *facto one*. The two major contenders were *Flow* from Facebook and *Typescript* from Microsoft. In terms of features, there didn’t seem to be a lot of difference, however, Flow was a bit ahead, having support for nice features like *async/await* and *decorators.* Not to mention, at the time, it did play better with React since it was birthed at Facebook.

Coming from a time when type-checking was not the in thing in the industry (frontend or backend), adding a type system to the MVP, irrespective of which, was already a bar raise in terms of code quality and maintenance. It is no doubt you’ve made the right guess, we did go with *Flow*.

As time crept by, Typescript began to demonstrate dominance in this sector. It presented a faster evolution, a much bigger community, quite a number of typed libraries, better support in plugins and IDE’s and many other advantages such as a widespread adoption within the npm ecosystem — which resulted in many (if not most) of the libraries having types publicly available.

On the other hand, we had begun to endure a lot of pain with the Flow checker in terms of endless analysis and memory leaks, and even though the Typescript checker was not a marvel in terms of performance, its granularity in relation to the number of flags during configuration had become really attractive. As you must have rightly guessed again, we decided to jump ship 😄 .

## Embarking on the big migration - The Dos and Don’ts

During an initial review of the intended approach, we listed a set of constraints to work within, in order to make the migration a success. The list included the following:

- **Freezing the code was not acceptable:** Imagine telling the development team, “hey 👋 , we want to run a crazy code migration which will last an undefined amount of time and as a result, you’ll have to pause work...”, unrealistic right? Stopping the development machine would have added too many risks to the project.
- **Git history needed to be preserved:** The source of the code is much more than the lines of code that comprise it. It is the history behind each one of those lines, and it provides the usefulness of navigation. It would have been a pity to lose it forever due to a code migration.
- **We didn’t want to interfere with pull requests in progress when the final migration happens:** Related to the two previous points, it was our intention to make the process easy in terms of open pull requests after the migration, so as to avoid too many conflicts and headaches.
- **A split-brain was not an option:**  Many tutorials explain the capability of having Typescript among JavaScript files, thereby adopting the process of a progressive migration. However, we wanted to prevent having a *split-brain* in our team- not knowing which technology to use at a certain point in time and elongating the migration for an undefined amount of time.

These constraints weren’t too difficult to overlook, but as you will see, we tried to apply them as much as possible, defining the long-term challenges of the migration.

## Caught Between Two - What do we do?

Prior to starting work on the migration, we researched other teams out there with the same challenge of migrating a huge codebase from Flow (or Javascript) to Typescript. We did learn a lot, and we took home some great points to consider before starting the migration.

This led us to two options as it regards the strategy to migrate the code:

- **Option A:** Migrate big areas of our codebase to minimize the split-brain problem. This would involve progressive migration, beginning with folders without dependencies.
- **Option B:** Create a “one-shot” script to migrate the whole codebase. This would be extremely difficult to get right, as it would effectively involve being able to translate every bit of Flow to TypeScript in an automated manner.

In the end, we decided on a mix of both approaches (well, who wouldn’t want the best of both worlds?). We found a big chunk of our code that was not coupled to the rest of the files and it became a nice target for the migration and validating the idea of having Typescript in production. Surprisingly enough, we began with this portion and its delivery was a success 🎉 — we had just validated that Flow and TypeScript could live side-by-side. As a result of this, we felt comfortable delivering the rest of the source code in one shot.

## Another Dilemma? - A new tool for the toolshed

- **[Ts-migrate](https://github.com/airbnb/ts-migrate)**: One of the most brilliant tools found was `ts-migrate` from Airbnb, which proposes to migrate Javascript codebases automatically, taking into account the Typescript Language Server output to decide what to do with each one of the offenses that appeared. We discarded this path because it was not fully aligned with the fact that we were migrating from Flow and not from a vanilla Javascript project.
- **[Flow-to-ts](https://github.com/Khan/flow-to-ts)**: One of the main tools adopted for this migration was `flow-to-ts` from Khan Academy. It was a fundamental piece for this migration, and we are thankful for all the work behind it. We'll talk more about why we chose this later.

## Very well, Shall we begin?

To abide by the most important constraint mentioned above (avoid a code freeze), we tried to merge as much innocuous code as possible into the master branch. By this, we mean pushing all the Typescript configuration files, typing definitions, etc.

On the other hand, we had a branch with a Bash script that had all the steps to proceed with the migration. Below is a summary of this script in steps:

1. Run a set of custom JsCodeShift transformations
2. Git rename all files from `js` and `jsx` to `ts` and `tsx`
3. Run the `flow-to-ts` transformation on all of them
4. Run linting and formatting tools to stabilize the format
5. Run the `tsc` validation and review the offenses

### 1. Custom JsCodeShift transformations

With the above steps in mind, we followed an iterative path of decreasing the number of existing offenses detected by `tsc`. For all the offenses that appeared repeatedly, we tried to code some [JSCodeShift](https://github.com/facebook/jscodeshift) transformations. We used [ASTExplorer](http://astexplorer.net/)  intensively as a playground to code our custom transformations. Seeing that we were coming from a Flow code base, we took advantage of it, and automatically tweaked the code related to types, in order to ease the job leading to `flow-to-ts` step.

In this step, we ended up with more than 10 custom transformations, which were boosted reducing all of them at the same time.

```jsx
module.exports = function (file, api, options) {
  const fixes = [
    transformation1,
    transformation2,
    transformation3,
		...
  ]

  return fixes.reduce((src, fix) => {
    if (typeof src === 'undefined') {
      return
    }

    return fix({ ...file, source: src }, api, options)
  }, file.source)
}
```

### 2. Git rename

Following the script, the next important step was the file rename. We did it with `git mv` to keep the Git history as much as possible:

```bash
find $@ -iname "*.js" | while read line; do git mv -- $line ${line%.js}.ts; done;
find $@ -iname "*.jsx" | while read line; do git mv -- $line ${line%.jsx}.tsx; done;
```

It is a simple change that can be easily missed but really significant in the long term.

### 3. Run flow-to-ts

On getting to this step, the script performed the main transformation, using the `flow-to-ts` package. This is a fundamental tool for Flow projects because it knows many Flow patterns that have a deterministic way to be mapped into Typescript. This is internally a huge JSCodeShift transformation, but with some preconfigured settings that make this tool easier to work with.

You can find a nice playground to understand how this package works [here](https://flow-to-ts.netlify.app/).

### 4. Linting and formatting

Although `flow-to-ts` uses [recast](https://github.com/benjamn/recast) underneath and tries to keep the original code formatting as much as possible, there are some previous transformations that create new code that never existed before. This code needed to be formatted to match our internal rules.

This step was a bit slow, but it was really worth it. Getting a result that was as close as possible to the original code was crucial to preserve the Git history.

### 5. Validate with `tsc` and review

Having this script working, the only missing step to complete the migration was to solve the rest of the offenses that were not repetitive. To be honest, this was the hard part, because we had to go through each one of them and resolve them. The cool thing about this process and having Flow in our codebase was that we were able to keep pushing type fixes into master without modifying the behavior of the application. Those fixes were properly handled by the iterative process after rebasing from master.

## What a ride! - Take-home lessons after a safe landing

Yes, we did it! Taking in account the first migration and the big one, we migrated 400k plus lines of code to TypeScript and boy! we did learn some valuable lessons.

![flow-to-ts-changes.png](flow-to-ts-changes.png)

 A few of them worth mentioning are:

- **Be transparent and communicative all the way, in order to get a buy-in from the team**. The worst possible scenario is doing an insane amount of work in misalignment with each one of the team members. We recommend that you share, involve, and document, each one of the steps, milestones, and key dates involved, instead of hoarding such information.
- **Stock up before embarking on the migration**. It is a long process that will require a deep understanding of your codebase, and you will probably find blockers that will force you to redo previous jobs in order to accommodate the last step. An example in our case was the usage of decorators that bothered the `flow-to-ts` step.
- **Try to picture where the code will stand after the migration and ask the right questions**. Would all the tools be ready after the migrations? What would happen with all those open pull requests? Will there be linting and formatting issues after the migration? For us, these kinds of questions were answered before the merge.
- **Do well to squash all the commits related to the migration into a single commit**. This will ease the path during the deployment and if something unexpected pops in, it is the best way to roll it back.

## Are we done? Not today! - Next steps

Our journey with Typescript doesn’t end here. In fact, it is just the beginning! After a few days of the team adopting the language, we decided to enable the `strict` flag in the compiler. Fortunately for us, we can do this gradually since you can include those flags one at a time.

We are also considering other tools and opportunities as well. For example, we currently employ the use of Webpack to build our application, but we have begun to consider [esbuild](https://github.com/evanw/esbuild) and [swc](https://swc.rs/) since both compilers have better support for Typescript than Flow. Be sure that we would keep you updated on whatever path we choose in this regard 🤞

## We did it - You can too! 💪

A few months ago, we made a bet on the usage of Typescript in our frontend codebase. We analyzed the pros and cons and traced a clear plan for such a big migration. Today, two weeks after, we can confirm that the team, delivers faster, and we have raised the quality threshold. This has ended up in fewer bugs in production and achieved the ultimate goal of happier customers.

We sincerely hope this post helps you if you are in the same situation as we were. 💪


How we migrated from Flow to TypeScript, the challenges faced and how they were surmounted in order to ensure a safe landing on the other side.


How we migrated 400K+ lines of code from Flow to TypeScript


Since a few years ago the idea of a [Second Brain](https://fortelabs.co/blog/basboverview/)
has been resonating with me. The capability to store an infinite amount of
information interesting to me and be able to sort it, categorize it and create
relations between items, lets you increase the knowledge in many topics in this
world and improve day by day.


After trying and evaluate many solutions out there like
[Workflowy](https://workflowy.com/), [Notion](https://workflowy.com/) and now
[Obsidian](https://obsidian.md/), it is clear to me that the best approach to
store this kind of information is simple Markdown files. The simplicity and the
capability to backup it somewhere is the main reason to follow this way.

But coming from Notion, I miss some nice features like kanban views, calendars, or
tables. Something that doesn't come out of the box in the Markdown spec.

One cool approach to store complex information in markdowns is by the block of
code prefixed by &#96;&#96;&#96;. You have seen this popular syntax in many
source control platforms to show syntax highlighting of different programing
languages.

## Goal

In this post, I want to explore and experiment with the possibilities of using this
approach to render more rich components to show plain text, like the one
required to model a simple interactive table.

The goal after interact with the UI is to be able to store back to plain text into markdown
files and close the cycle. This idea is not new and it is implemented in other places like
[Github todos](https://docs.github.com/en/github/managing-your-work-on-github/about-task-lists),
[kanbans](https://marketplace.visualstudio.com/items?itemName=coddx.coddx-alpha)
or [complex UML diagrams](https://plantuml.com/). To shine with this small
project, I wanted to play with some technology new to me like
[WASM](https://webassembly.org/).

To understand the project vision you can check the
conceptual lifecycle of the information in the followin diagram:

![Markdown UI lifecycle](markdown-ui.png)

Also, I wanted to refresh my knowledge about lexers and parsers, taking the
opportunity to implement them in [Rust](https://www.rust-lang.org/).

## First step: Parsing

To be able to validate the idea and achieve the minimum viable product, we will
take the easiest example to play with. We are going to build an interactive
markdown table. All values in this table will be floats. This is an example of
the content:

````
```table
2.3,6,9
12,3.44,5
9,2,2
```
````

First of all, we need a grammar, similar to CSV structure, and we are going to
use [Parsing Expression Grammar](https://en.wikipedia.org/wiki/Parsing_expression_grammar)
(PEG). This grammar could be implemented in
Javascript with a library like [pegjs](https://pegjs.org/), but since we are
implementing this step in Rust, we are going to use
[pest-rs](https://bitbegin.github.io/pest-rs/) crate.

The grammar needs to support floats for each one of the cells, called in this
case `Field`. Then, a list of floats separated by commas will form the rows of the
table, in this case, called `Record`. Then a `File` will be a list of `Record`
separated by break lines:

```peg
Field = { (ASCII_DIGIT | "." | "-")+ }
Record = { Field ~ ("," ~ Field)* }
File = { SOI ~ (Record ~ ("\r\n" | "\n"))* ~ EOI }
```

We need to understand that this grammar doesn't include the classic Markdown code
_fence_ (&#96;&#96;&#96;) because this responsibility belongs to the parent
Markdown parser.

Once we have a nice grammar definition, we can use it to build the parser.
Pest is going to build all the utility modules to be able to parse the incoming
data to abstract syntax tree nodes.

First, it generates the required `Enum`'s to be able to differentiate the
parsed tokens.

```rust
pub enum Rule {
    EOI,
    Field,
    Record,
    File,
}
```

On the other hand, it will also generate a parser to build the correct tree of
nodes:

```rust
extern crate pest;
#[macro_use]
extern crate pest_derive;

use std::fs;
use pest::Parser;

#[derive(Parser)]
#[grammar = "csv.pest"]
pub struct CSVParser;

fn main() {
    // A sample how to parse a cell / field from our grammar:
    let successful_parse = CSVParser::parse(Rule::Field, "-273.15");
    println!("{:?}", successful_parse);

    let unparsed_file = fs::read_to_string("./samples/test.csv").expect("cannot read file");

    let file = CSVParser::parse(Rule::File, &unparsed_file)
      .expect("unsuccessful parse") // unwrap the parse result
      .next().unwrap(); // get and unwrap the `file` rule; never fails

    // Print the parsed abstract syntax tree
    println!("AST: {:?}", file);
}
```

If we run this code with the `test.csv` as input, we are going to get:

```rust
Pair {
	rule: File,
	span: Span { str: "2.3,6,9\n12,3.44,5\n9,2,2\n", start: 0, end: 24 },
	inner: [
		Pair {
			rule: Record,
			span: Span { str: "2.3,6,9", start: 0, end: 7 },
			inner: [
				Pair { rule: Field, span: Span { str: "2.3", start: 0, end: 3 }, inner: [] },
				Pair { rule: Field, span: Span { str: "6", start: 4, end: 5 }, inner: [] },
				Pair { rule: Field, span: Span { str: "9", start: 6, end: 7 }, inner: [] }
			]
		},
		Pair {
			rule: Record,
			span: Span { str: "12,3.44,5", start: 8, end: 17 },
			inner: [
				Pair { rule: Field, span: Span { str: "12", start: 8, end: 10 }, inner: [] },
				Pair { rule: Field, span: Span { str: "3.44", start: 11, end: 15 }, inner: [] },
				Pair { rule: Field, span: Span { str: "5", start: 16, end: 17 }, inner: [] }
			]
		},
		Pair {
			rule: Record,
			span: Span { str: "9,2,2", start: 18, end: 23 },
			inner: [
				Pair { rule: Field, span: Span { str: "9", start: 18, end: 19 }, inner: [] },
				Pair { rule: Field, span: Span { str: "2", start: 20, end: 21 }, inner: [] },
				Pair { rule: Field, span: Span { str: "2", start: 22, end: 23 }, inner: [] }
			]
		},
		Pair { rule: EOI, span: Span { str: "", start: 24, end: 24 }, inner: [] }
	]
}
```

## Second step: Render

Now that we have the required structures to keep the AST of the table, we are
going to send them as objects to Javascript world. Those objects will be the
source that will feed some UI built on top of libraries like [React](https://reactjs.org/)
or [Vue](https://en.wikipedia.org/wiki/Parsing_expression_grammar). In
this case, we are going to follow the React path.

Those [DTO](https://en.wikipedia.org/wiki/Data_transfer_object)'s will be
represented by Rust structs that will hold the information parsed in previous
steps. To be able to go from WASM to Javascript, we need to add `Serialize` and
`Deserialize` traits to our structures.

```rust
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
pub struct Field {
    pub value: f32,
}

#[derive(Serialize, Deserialize)]
pub struct Record {
    pub fields: Vec<Field>,
}
```

Thanks to [wasm-bindgen](https://rustwasm.github.io/wasm-bindgen/) and
[wasm-pack](https://github.com/rustwasm/wasm-pack), we are able to build a
package that includes a declaration of a `parse` function that can be called
from Javascript world:

```rust
#[wasm_bindgen]
pub fn parse(source: &str) -> JsValue {
  let file = CSVParser::parse(Rule::File, source)
    .expect("unsuccessful parse") // unwrap the parse result
    .next().unwrap(); // get and unwrap the `file` rule; never fails

  let mut records = vec!();

  for record in file.into_inner() {
    match record.as_rule() {
      Rule::Record => {
        let fields = record.into_inner().map(|field| {
          Field {
            value: field.as_str().parse::<f32>().unwrap()
          }
        }).collect();

        records.push(Record {
          fields
        })
      }
      Rule::EOI => (),
      _ => unreachable!(),
    }
  }

  JsValue::from_serde(&records).unwrap()
}
```

This function will return `JsValue`'s with a representation of our `Field` and
`Record` structures. Those objects will be used to render a nice and
interactive UI build on top of React.

```javascript
import React from 'react';

const App = () => {
  // Retrieve it from markdown
  const source = "2.3,6,9\n12,3.44,5\n9,2,2\n"

  const [parse, setParse] = React.useState(() => {})
  const [records, setRecords] = React.useState([])

  React.useEffect(() => {
    import('./csvlib').then(({ parse }
    ) => {
      setRecords(parse(source))
      setParse(() => parse)
    })
  }, [])

  return (
    <div className="App">
      <table className="table-auto border-collapse border border-green-800">
        <tbody>
          {records.map((record, index) => (
            <tr key={index}>
              {record.fields.map((field, index) => (
                <td key={index} className="border-2 border-green-600 px-6 py-2">
                  {field.value}
                </td>
              ))}
            </tr>
          ))}
        </tbody>
      </table>
    </div>
  );
}
export default App;
```

## Third step: Interaction

Now that we are able to render the parsed content, we can continue adding a bit
of interactivity to the view. You can imagine whatever nice use case for your
widget. From adding buttons, drag and drop, calendars, etc.

For demo purposes, we are going to add just a button to add new rows to
the table.

![](table.png)

This button will modify the internal state of Javascript objects that represents those
records and the UI will react accordingly. As an example, we are going to as to the user
a new row of float numbers to be added to the table:

```javascript
  const onClick = () => {
    const value = prompt('Add a new row. Floats separated by commas:')
    const items = parse(`${value}\n`)

    setRecords([...records, ...items])
  }

  ...
  <button onClick={onClick}>
    <span>Add new row</span>
  </button>
```

## Fourth step: Serialization

Finally, the last step in this cycle is to serialize the React state back to
Markdown plain text. To do this, we will call again a WASM function to
compute this. With this approach, we encapsulate all the logic about parsing and
serializing in the same WASM module. To achieve this, we are going to code the
`store` function and expose it to `wasm_bindgen`:

```rust
#[wasm_bindgen]
pub fn store(js_objects: JsValue) -> JsValue {
  let records: Vec<Record> = js_objects.into_serde().unwrap();

  let value = records
    .iter()
    .map(|record| {
      record
        .fields
        .iter()
        .map(|field| {
          field.value.to_string()
        })
        .collect::<Vec<_>>()
        .join(",")
    })
    .collect::<Vec<_>>()
    .join("\n");

  JsValue::from_serde(&value).unwrap()
}
```

Then, we call this new function from React passing as argument all rows in the
table from the state:

```javascript
import React from 'react';

const App = () => {
  ...
  const [store, setStore] = React.useState(() => {})

  React.useEffect(() => {
    import('./csvlib').then(({ parse, store }
    ) => {
      ...
      setStore(() => store)
    })
  }, [])

  const onRetrieve = () => {
    const source = store(records)
    const markdown = `\`\`\`\n${source}\n\`\`\``

    console.log('Markdown:')
    console.log(markdown)
  }

  return (
    ...
  );
}
export default App;
```

And we get the final goal of new Markdown 🎉 for this dynamic table component:

````
```table
2.3,6,9
12,3.44,5
9,2,2
1,2,3
```
````

## Conclusion

We went through all the steps required to parse arbitrary data to feed a reactive
UI and close the cycle back to text. As we said, the possibilities of this
approach for markdown documentation or more rich second brains are endless.

To see a working demo widget, check the widget below:

<TableDemo />

You can find the demo source code [here](https://github.com/fcsonline/markdown-ui-table)

In a next post we are going to implement an interactive kanban board with tasks
following the same strategy as this one. Stay tuned!

Thanks for reading me and to all people that helped me to polish this article!

Happy hacking!


An experiment on top of WASM about how to build rich interactive components based on Markdown content.

Markdown UI


Everybody is familiar with the
[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle in
software engineering coined by Andy Hunt and Dave Thomas. It states *do not
repeat yourself* in code to achieve less ambiguous code, avoid redundancy, and
many other benefits in the long term.

In this post, I want to explain another pattern that I see over and over in all
kind of projects and it brings problems and impacts the
maintainability of your software project. It doesn't affect the source code
itself but the organization of the files that compose it.

## What is a leftover?

In the lifespan of a software project, code gets created and destroyed to meet
users' expectations. During this period, it is easy to generate files,
components, and other kinds of stuff that are not used any more. Those
undesired pieces of software, make it harder to maintain, impact on your
internal metrics, slow down your Continuous Integration check and other kinds
of side effects.

Those leftovers from the past, don't bring any value to the project and it can
get to the point where nobody is brave enough to do a right cleanup to have a
healthy codebase. This goes directly to the technical debt bucket.

Those items should have been removed when the last reference to them
disappeared but a developer forgot to remove them. Those leftovers can be a
unused image, a unused copy, a unused test on your test suite, etc.

You can find many similarities and parallelisms with the responsibilities of a
memory garbage collector. It keeps tracks the dependencies between parts, with a
[reference counting](https://en.wikipedia.org/wiki/Reference_counting) to be
able to destroy all unused references. The problem here is that we don't track
this kind of information between the dependencies in our code base.

## Why do we create leftovers?

This bad design pattern can have many shapes. This is another example
where *Don't create leftovers* impacts many projects today. How many of you
have a translation system where translations live in one place and the place
where they are used is in another one?

`src/translations/en.json`
```json
{
  "hello_world": "Hello World!"
}
```

`src/components/hello_world.jsx`

```js
export default function HelloWorld () {
  return (
    <div>{i18n.t('hello_world')}</div>
  )
}
```

The problem with this translation approach is that if the `HelloWorld`
component disappears in the future, the copy in `en.json` will be forgotten unless
a developer remembers to clean it up. At some point in the future, your
main translation file starts to have hundreds of unused keys that make you lose
money with each new language you want to support.

![Example of translation dependencies](dependencies.png)

Some [translation systems](https://formatjs.io/docs/getting-started/message-extraction)
are aware of this and follow a much better approach. They have an *extraction*
step to retrieve all translations from the codebase. With this, the list of
used copies (dependencies) is always up to date.

## Other examples

How many times have you seen a web platform tutorial where the `assets` folder
and the place where they are used is far away?

Take this easy file structure:

```
/assets/images/
  - post1-cover.jpg
  - post1-thumbnail.jpg
/posts/post1/
  - helper.js
  - index.jsx
/specs/
  - helper.spec.js
```

This structure can be familiar to you, but it suffers from DCL. If for some
reason the `post1` is no longer needed and it disappears, it is really
probable that the `helper.spec.js` file will be removed after a failing test but
those `assets` related to `post1` will be forgotten forever.

Check this other example:

```
components/
  List/
    index.jsx (has a dependency with ListItem)
  ListItem/
    index.jsx
stories/
  List.story.jsx
```

Although this file organization seems inoffensive, it suffers from DCL too. If
the `List` component disappears, maybe the `List.story.jsx` will fail as a
side-effect but nobody will alert you that `ListItem` needs to
be removed too.

## How to detect it?

This principle shows up like references to parent components or external
resources, but sometimes it is harder to detect it.

The rule of thumbs in all cases is to answer yourself the question:

*If I remove this component or piece of code from my code base, will I create leftovers?*

With this question, you imagine yourself in the future removing this shiny new
code and seeing all the unused pieces in your source code. Answering this
question at this point helps you to know all the dependencies this code has.

## Better approach

To fix this organization problem, the best approach is to encapsulate better
all the dependencies. In the case of file structures try to avoid having
dependencies to parent components if a dependency is only used once and move
them as child components. All those dependencies can start to use relative
paths.

Following the previous example, this would be a much better organization:

```
/components/
  List/
    ListItem/
      index.jsx
    index.jsx
    index.story.jsx
```

This new approach has much better encapsulation and a clear set of
dependencies. One direct benefit of this is that it lets you move this component
to others around without breaking it.

In some cases, like the translations example, you will need some extra step in
your toolchain to resolve those dependencies of your views. In the case of the
assets one, you will need your builder (Ex. Webpack) to have support to resolve
them.

## Conclusion

We just reviewed a few examples of the *don't create leftovers* principle.
Understanding the consequences in terms of maintainability, you will be able to
create and organize your components in a better way that will let you
maintain your code base in a healthier way in the long term.

You don't need to take this principle as written in stone. There are always
exceptions or edge cases, as in DRY principle, we don't have to fall on *the
wrong abstraction* as Sandi Metz mentioned. You have to use it and keep it
in mind when you structure your components, views, assets or
whatever piece in your project.

I hope this post helps you to spot this pattern sooner rather than later to
keep the maintainability of your project under control.

Happy hacking!


Improve the maintainability of your projects detecting this bad pattern in your code.


Don't create leftovers (DCL)


I remember it was about two months ago when I was talking with Eduardo
Lanchares about the best stack for the next webapp. Sometimes you are at work
and you are stuck with the current set-up of frameworks and tools that are
quite difficult to change or refactor. This forces you to always work with the
same mentality not allowing you to test new approaches to solve problems.

Following our desire to learn new stuff to make our life easier, we wanted to
experiment with new technology to build a side project with a complete new
stack that fits better with the app requirements.

In that same week, a friend of mine was talking about a bug that was affecting
him in the web client of Transmission. This [bug](https://trac.transmissionbt.com/ticket/5857)
was affecting more people because we found an open bug report in the tracker
that was 2 years old. We decided to check the source code and try to fix it
because we thought that would be a piece of cake. But a few hours later we
discovered that was not as easy as we expected and at some point we gave up.

With all these things in mind we had the perfect scenario and excuse to start
playing with a new side project. We decided to start building a new interface
from scratch to understand new patterns, new approaches and discover what would
be the next challenges for the _ultimate interface_.

## Our experimental stack

Reviewing all of the features in the current Transmission web client, the stack
we chose was React for the user interface, Mobx for the state management and
CSS modules for all the stuff related with the application styles, with a new
build process managed by Webpack.

On the other hand, and with the above stack in mind, we decided to use some
best practices for the long term quality of this project. Some of these were
things like a strict set of Eslint rules, unit tests, UI tests and
internationalization.

With this decision we wanted to achieve a more reliable interface, better
performance, security and correctness. In conclusion a better way to maintain
this web application.

In the following sections we are going to go through a list of differences
between the way of developing the original Transmission source code and our
approach. If you are curious, you can check the [original
code](https://github.com/transmission/transmission/tree/master/web) and [our
code](https://github.com/fcsonline/react-transmission).


## Components

In terms of application modularization, the original client was implemented
with a set of _views_ joined in a single static HTML file. These views were
modified by jQuery code to keep it updated. While this approach worked as
expected, it had the disadvantage of generating multiple DOM updates, which
impacted UI performance. Also, having hard-coded strings in this file made it
difficult for supporting I18n as well as dynamic content generation (for
instance when displaying available [speed limits](https://github.com/transmission/transmission/blob/master/web/index.html#L380))

Our new application stores each component in an independent folder for each of
them, with all the related stuff within. Things like styles, tests or
subcomponents live together. This lets us manage all the application components
understanding all the dependencies between them and achieving the best
decoupling possible.


## State management

In a traditional web application implemented with jQuery you would keep
updating all the UI manually, after realizing that some elements were outdated.
Those changes would usually come from some user interaction or from server
updates, so the concept of application state was almost inexistent.

With the new stack managed by React and Mobx, we follow the one way data flow
convention, which makes it easier to reason about the application state. All
this state is managed in Mobx containers called stores. This is a really good
diagram to understand how it works:

Mobx uses the following main concepts to model your application state:

- `observable` properties: basic tracked properties
- `computed` properties: derived properties from observable ones
- `actions`: methods to update the state
- `reactions`: side effects to be executed for each state change

For instance, in the following example, when we click on the preferences
button, we dispatch an action that updates the `isPreferencesDialogShown`
observable property. That change produces an update in observer components. In
this case `PreferenceDialog` reacts to this change and re-renders itself,
resulting in dialog shown in the UI.

![Inject stores to views](inject-store.png)

## Styling

For a really long time we have been styling HTML components through a list of
CSS classes referenced by class attributes in your elements. New tools have
appeared during the last 5 years to improve our life like SASS, LESS or CSS
Modules.

If you are used to code big sets of components you can make the common mistake
of reuse style classes for different components. But what happens when the
styles from those components start to differ? You are in a trouble.

As we said before, in our stack we decided to use CSS Modules to style all our
components. We think that this is the best approach to achieve a modular and
reusable set of classes. All the required classes for each component live
together. This is really good in terms of maintainability because it’s trivial
to decouple components. CSS Modules lets you play with composition so you get
the best characteristics from both worlds.

As you can see in the following example we are using two CSS modules references
called `content` and `logo`. These names are used to refer classes in the
stylesheet file.

React `viewConnectionDialog/index.js`:

![Connection dialog with styles](connection-dialog-js.png)

Style `fileConnectionDialog/styles/index.css`:

![CSS code for Connection Dialog](connection-dialog-css.png)

## Tests

The original application had no tests at all so it was easy for us to improve
that aspect.

In the new application we wanted to test not only business logic but also the
UI. For this we opted for the [Jest](https://facebook.github.io/jest/)
framework and the _snapshot_ testing technique. With this we ensure every
component renders what’s expected. This is important because most of the
components contain several paths and we need to ensure that each one is
covered.

Integrating this new set of tests to some CI we can be more confident about
future changes that we apply to the application.

As we can see in this example we are rendering the Progress bar component with
mocked torrent data.

![Jest progressbar tests](progressbar-test.png)

Then with the snapshot technique we can ensure that the rendered result is what
we expect comparing with previous snapshot. Testing components in this way is a
as simple as record JSX code.

![Jest progressbar snapshots](progressbar-snapshot.png)

## Extra features

Reviewing all available features in the original Transmission web interface we
discovered broken features like Desktop notifications. We wanted to restore
them because we think it’s a really nice feature to have.

Furthermore, one extra feature that the new interface includes is drag and drop
file support for torrent files. This boosts the common use case of quickly
adding a torrent file as you can see in this example:

![A small React Transmission demo](react-transmission-demo.gif)

## Conclusion

We started this project to play with new technology to be able to understand
all the challenges a bigger project would face if you chose some specific
stack. At some point we saw how easy was to implement new features that were
included in the original interface. So we decided to push forward and set a
personal goal: Implement all of them. Today we can say that we’ve achieved 95%
of all features.

Our last goal is to present this project to the community to be evaluated. We
decided to respect as much as possible the original interface for two main
reasons:

- We didn't want to raise a debate in UI design. So we decided to follow the current design, we just wanted to propose a new code architecture.
- We didn't want to impact the end users in case that this code were considered for inclusion in the main Transmission repository.

Lastly, even if the project dies out, for whatever reason, we’re happy anyway,
because we learnt a lot of things and had a really fun time during its
development. :D



A walkthrough explaining the process to rebuild the web interface of Transmission with a modern stack, replacing old jQuery functionality with React and Mobx libraries.


Rewriting Transmission UI with React

![Picture from Amazon.com](game-wood.png)

I'm sure that you played this game hundred of times when you were a kid. At
that time you learned how to fit a different kind of shapes in their
corresponding places. This game is going to be a good metaphor for the next
story.

I'm going to go through one example to know how to identify you are mixing
responsibilities between components and how to enforce the [single responsibility principle](https://en.wikipedia.org/wiki/Single_responsibility_principle).
When we are going to highlight the figures of what is content and what is a
container.

Falling into this bad pattern of mixing responsibilities can impact in many
components in your source code not only in your component style but even in
your component logic. The main goal is to get better components in terms of
reusability. I'm going to use this example to analyze it in terms of:

- Components styling (UI)
- Component business logic (UX)

All the examples presented in this post are based on React classes and JSX markup but those advice apply for whatever technology you use.

## List of cart items

Imagine a list of items in your cart. We have three items in our cart, two of
them are selected and one of them is a _premium_ item. Each item has a button
to let the user remove items from the cart.

An easy example for it could be something like this:

```html
<ul class="cart">
  <li>
    <header>Item 1</header>
    <p>Lorem ipsum dolor sit amet, consectetur adipiscing.</p>
    <button>Remove from cart</button>
  </li>
  <li class="premium selected">
    <header>Item 2</header>
    <p>Fusce vel neque sit amet felis fermentum cursus vitae.</p>
    <button>Remove from cart</button>
  </li>
  <li class="selected">
    <header>Item 3</header>
    <p>Vel neque sit amet vitae suscipit nibh.</p>
    <button>Remove from cart</button>
  </li>
</ul>
```

And the stylesheet something like this:

```css
.cart {
  border: 1px solid #ddd;
}

.cart li {
  margin-bottom: 10px;
  border: 1px solid red;
}

.cart li.selected {
  background-color: blue;
}

.cart li.premium {
  font-weight: bold;
}
```

When you start trying to convert this layout in components the first approach
that comes to your mind is something like:

```jsx
class CartList extends React.Component {

  onClickItem(event, index) {
     this.state.setState({
        selected: [
           ...this.state.selected,
           index
        ]
     });
  }

  onClickRemove () {
    // Go to your store and remove item
  }

  renderCartItem(item, index) {
    const isPremium = item.premium ? 'premium' : '';
    const isSelected = this.state.selected.indexOf(index) >= 0 ? 'selected' : '';

    return (
      <li className={`${isPremium} ${isSelected}`} onClick={(event) => this.onClickItem(event, index)}>
        <header>{item.title}</header>
        <p>{item.description}</p>
        <button onClick={this.onClickRemove}>Remove from cart</button>
      </li>
    );
  }

  render() {
    const { items } = this.props;

    return (
      <ul className='cart'>
        {items.map((item, index) => this.renderCartItem(item, index)}
      </ul>
    );
  }
}
```

_An example of mixed responsibilities_

This list will be the `CartList` component and the `renderCartItem` is going
to render `li` nodes with all the required stuff for each item in the list
with all the required classes to show all the selected items.

Maybe at that point you don't see the problem but sooner than later you will
start facing problems regarding this decision.

A few weeks later, you get new requirements for your cart. Users should be able
to multiselect items in this list. You start implementing this feature and you
start needing some way to communicate between items to know if they are
selected, sending events, toggling classes in the item, etc.

Wait a minute, this is going crazy…

If you reuse those items in another place in your application, maybe you don't
want the selection behaviour and you start adding more and more conditionals to
choose which are the behaviours you want for all the places they are used. Do
you want to add the selectable behaviour to a list of Order, but this one is
`CartItem` types?

So, the answer for including this behaviour in the item is probably no.

Now, stop for a second and think in terms of containers and content. Which are
the container and the content for this layout?


### New list of cart items

It's really good when you start building a structure of components to think
about which is the responsibility for each one in terms of reusability.

Differentiate content from containers. Ask yourself:

- Are those items responsible for managing the selection in this list?
- Which is the best place to put a handler for managing the selection?

It's easy to change your mind and start seeing them from another point of view.
Imagine a component `SelectableList` and `CartItem`. Let's check a new approach
with this layout:

```html
<ul>
  <li>
    <div>
      <header>Item 1</header>
      <p>Lorem ipsum dolor sit amet, consectetur adipiscing.</p>
      <button>Remove from cart</button>
    </div>
  </li>
  <li class="selected"> 
    <div class="premium"> 
      <header>Item 2</header>
      <p>Fusce vel neque sit amet felis fermentum cursus vitae.</p>
      <button>Remove from cart</button>
    </div>
  </li>
  <li class="selected">
    <div>
      <header>Item 3</header>
      <p>Vel neque sit amet vitae suscipit nibh.</p>
      <button>Remove from cart</button>
    </div>
  </li>
</ul>
```

_New cart markup, with better responsibilities_

The differences are subtle, but you will start taking advantage of them.

![A better approach for containers](new-cart-items.png)

### Components styles

A common pattern to know you are mixing responsibilities between components is
if you are abusing of `margin` CSS rule. Content components should expand as
much as possible and should not be aware of the surroundings. Container
components are the right ones for managing the space between content
components, and you can achieve that with `padding` and `gap` CSS rules.

```css
.selectable-list {
  border: 1px solid #ddd;
}

.selectable-list li {
  padding-bottom: 10px; /* margin to padding ;) */
}

.selectable-list .selected {
  background-color: blue;
}
```

As you can see we moved the `margin-bottom` rule from the cart item to a
`padding-bottom` rule in `selectable-list`.

**Try to convert component margins to paddings**

When you start using this approach you will discover that specs like [flexbox](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Flexible_Box_Layout/Using_CSS_flexible_boxes)
fit much better with all your requirements.

### Components business logic

One of the most interesting features from DOM and of the most misunderstood is
the event bubbling. When you understand how it works and which are the main
topics you need to take care of is when you all this new component structure
shines.

A brief summary for event bubbling is that an event that happens in a DOM node
goes through all the parent nodes until you call `stopPropagation` method.

![An event propagation example](event-propagation.png)

Taking this as premise, it's really easy to share all the responsibilities out
in our application between all the components in it.

If you think about who is the responsible for managing the selection, now the
answer is really easy. It must be the `SelectableList`.

**SelectableList:**

```jsx
class SelectableList extends React.Component {

  onClickItem(event, index) {
     this.state.setState({
        selected: [
           ...this.state.selected,
           index
        ]
     });
  }

  render() {
    return (
      <ul>
        {items.map((item, index) => {
          const isSelected = this.state.selected.indexOf(index) >= 0 ? 'selected' : '';

          return (
            <li key={index} className={isSelected} onClick={(event) => this.onClickItem(event, index)}>
              {renderItem(item)}
            </li>
          );
        })}
      </ul>
    );
  }
}
```

**CartItem:**

```jsx
class CartItem extends React.Component {
  onClickRemove () {
    // Go to your store and remove item
  }

  render() {
    const { item } = this.props;
    const isPremium = item.premium ? 'premium' : '';

    return (
      <div className={isPremium}>
        <header>{item.title}</header>
        <p>{item.description}</p>
        <button onClick={this.onClickRemove}>Remove from cart</button>
      </div>
    );
  }
}
```
_CartItem component knows nothing from SelectableList component_

The really cool thing about this is that now the responsibilities are really
clear, and both components can evolve without clashing between them. Do you
want to add multiselection in `SelectableList`? Go for it. Are you adding new
stuff to `CartItem`? No problem.

In terms of component logic, play with event bubbling to decide who is the
owner for each event is generated in the application. If you think an event
should not go through upper components, stop it with `stopPropagation` method.

If you find yourself trying to disable some specific zone in your component,
you can take advantage of [`pointer-events`](https://developer.mozilla.org/en/docs/Web/CSS/pointer-events)
property to have a better control over what is clickable.

## Conclusion

You can extrapolate those examples to hundreds of use cases. The result will be
a really good set of reusable generic components and a set of easy components
linked to your business logic.

With this set of generic components, is easier to apply improvements to
accessibility in your application. This will step your application up in terms
of quality adding value to many parts of it at once.

Try to think in containers and content every time you are creating new
component and you will be able to figure out the best for you!

Learn how to split UI components and distribute their behaviour in a way that will let you build better and more maintainable interfaces.


Content vs Container


I have been a Vim user for 12 years and one important thing that you learn the
first days using it is that you can be super efficient typing commands to
complete what you are trying to do.

I spend almost all my day coding in front of a console and at some point you
realize that not only you can DRY your code, but your daily console commands
also. From time to time, I really like to stop for a while and think about all
the commands I have typed and see if I can execute them faster.

Few months ago, a friend of mine was trying to polish his daily pain points too
and he created this awesome plugin for tmux called tmux-fingers but I decided
to take another path. At the same time I discovered
[FZF](https://github.com/junegunn/fzf) and I thought it would
be an interesting tool to play with.

This command-line fuzzy finder does one thing and does it well, just like the
most ancient commands in UNIX. It takes an input and you can filter it to
choose what you want in a interactive shell. So you can do things like this
example:

[![demo](https://asciinema.org/a/22tnlseff7yuw62fc73nfelns.png?ts=1)](https://asciinema.org/a/22tnlseff7yuw62fc73nfelns?autoplay=1)

I recommend to you to play with it a bit and start thinking about all the
possibilities you can do with it. Review all the available FZF options and you
will discover gems like multiselection (-m).

So, how do you shuffle FZF and Git to get this posts’ topic answered? Think
about common git use cases that you do every day and you are going to think of
things like this:

- Switch between branches
- Review some specific commit
- Do interactive rebases
- Edit changed files in a commit

We are going to get our hands dirty with one example:

`git log --pretty=oneline | fzf`

After executing the command below, you realize that you can search for a commit in you repository in an interactive way. If you press ENTER, the FZF’s output will be the selected line you chose. Now, try to execute this:

`git show $(git log --pretty=oneline | fzf | cut -d=' ' -f1)`

And you will get you first interactive git show. Hooray! :p

## Use cases

Now take a breath and go through those use cases I detected in my daily workflows that apply to you. All my fuzzy git commands start with pam.

### Switch between branches

This is a classic use case. You want to switch to some branch where you have been recently. I execute this:

`git pam`

Internally what this command is doing is:

`git checkout $(git recent | fzf)`

[![demo](https://asciinema.org/a/15lmglaf6ojo44jo0fbv4tt6c.png?ts=1)](https://asciinema.org/a/15lmglaf6ojo44jo0fbv4tt6c?autoplay=1)

### Edit uncommitted files

This is another classic use case. You want to edit some non committed file you
are working on. I execute this:

`git pamvim`

Internally what this command is doing is:

`vim $(git status -s | fzf -m)`

This command supports multiselection!

[![demo](https://asciinema.org/a/7fbnjgoh41bg2ek9un5rxgn3x.png?ts=1)](https://asciinema.org/a/7fbnjgoh41bg2ek9un5rxgn3x?autoplay=1)

### Interactive rebase

This is getting interesting…

You want to start a interactive rebase from some specific commit and you want
to select it. I execute this:

`git pamrebase`

Internally what this command is doing is:

`git rebase -i $(git log --pretty=oneline | fzf)^`

[![demo](https://asciinema.org/a/b8xhvbr2rpw0cnjr2ga0dh0kh.png?ts=1)](https://asciinema.org/a/b8xhvbr2rpw0cnjr2ga0dh0kh?autoplay=1)

### Edit files in a commit

Another useful command. You want to edit some files that were changed in some
specific commit and you want to select it. I execute this:

`git pamvimlog`

This command supports multiselection!

[![demo](https://asciinema.org/a/c0y4ywx3wj5i8wh2epf08a51a.png?ts=1)](https://asciinema.org/a/c0y4ywx3wj5i8wh2epf08a51a?autoplay=1)

### Interactive fixup

And my last example. You want to do a fixup in some specific commit and you
want to select it. I stage some changes and then I execute this:

`git pamfix`

[![demo](https://asciinema.org/a/699sr7ume88wemx9p57n0ipr4.png?ts=1)](https://asciinema.org/a/699sr7ume88wemx9p57n0ipr4?autoplay=1)

## Conclusion

If you are interested in more useful commands like those, check my
[dotfiles](https://github.com/fcsonline/dotfiles/blob/master/git/gitconfig#L41)
out.

Happy hacking!


Improve your daily Git interactions with this awesome set of command line aliases, powered by FZF. You will be able to switch, rebase and fixup commits in a blink of an eye.


~ferran.basora

Welcome, I'm Ferran Basora

Recent posts

How we merged our main repositories into a single monorepo

How we migrated 400K+ lines of code from Flow to TypeScript

Markdown UI