jeremarc

Seek the Way, Not the Win

Sat, 29 Nov 2025 00:00:00 GMT

"Work and sports have many things in common. In Japan, competition is traditionally individual, as in sumo wrestling, kendō swordsmanship, and judo. In fact, in Japan we do not "compete" in these activities but rather we "seek the way and study it" devotedly. This approach has its analogy in the work arena, where the art of the individual craftsman is highly valued." — Ōno Taiichi, Toyota Production System (1978)

This means that a kendō duel does not have the goal of defeating an opponent. This duel is a way for you to assess how far you have come on your journey to kendō mastery. <strong>The result of such a duel is feedback you need to take your next step</strong>.

Failure is therefore your inability to see the gap in your own skills by focusing instead on the event that brought light to this gap.

You can only reach Mastery when Mastery is the goal, not the wins along the way, because let's face it, each small win or loss is only a point:

<figure class="video-wrap"> <video src="/videos/2024_dartmouth.mov" controls playsinline /> <figcaption>Roger Federer holds court at Dartmouth Commencement, 2024</figcaption> </figure>

Stress is Good

Tue, 23 Sep 2025 00:00:00 GMT

Stress management is not about reducing all kinds of stress.

There is a healthy amount of stress that gives you a sense of urgency towards specific outcomes. It pushes you from that inert state to do something.

People who deeply care don't solely rely on extrinsic motivators.

So many people avoid stress to an extreme where any level of urgency is regarded as "pushy" or "not sustainable". Staying in bed all day is not sustainable, so is excessive daily workouts. You will only be a healthy individual through balance.

Your first workout after getting out of bed won't feel good, and that's not feedback to stop working out, but an invitation to explore a new balance.

PThD

Thu, 04 Sep 2025 00:00:00 GMT

<div class="poem"> The panic hits at half past three<br /> "I'm late! At home I should be!"<br /> I rush back home with racing heart,<br /> Then freeze... there's no unfinished part.<br /> Months have passed since my defence,<br /> But still my body stays so tense.<br /> These phantom deadlines haunt my brain<br /> The PhD's lingering pain. </div>

Retreat to Win

Mon, 18 Aug 2025 00:00:00 GMT

"Without (leverage to change) policy, your tool is stepping back and allowing the brokenness to collapse under its own weight. A deeply flawed system can't be saved by Band-Aids, but it can easily absorb your happiness to slightly extend its viability." – Will Larson, Systems of Engineering Management [Sec 5.6.4] (2019)

What's sad about it is people actively not caring about saving some systems.

"Not my circus, not my monkeys" becomes the default motto.

Sometimes it's a character. They just couldn't care less about anything that is "not their job description". Mostly, it's a bad environment.

It's a signal of a deeply flawed reward system of the environment / management. Reward the superhero who saves the day after* shit happens. The ones who work tirelessly to avoid shit from happening? Meh, they're doing their job anyways. How to even prove it would have gone to shit? It takes a certain amount of skill for managers to differentiate between those two situations.

Very similar skills are required to hire people who care for the right reasons and not burn them out (See my post about hiring).

If you couldn't do the latter, you will fail at the former.

A deeply flawed system can't be saved by Band-Aids. That holds when you are convinced it's "deeply flawed", and you don't have the right authority to change things. You would be surprised how many times you will be given authority (if you ask for it) because no one knows what to do. Take a chance once in a while to learn what it takes to recognize and fix such a system and the importance of everyday tasks to avoid it.

Notes on How to get worse at programming (Andreas Kling)

Mon, 21 Jul 2025 00:00:00 GMT

I just found a video by Andreas Kling from 6 years ago (2019.07.23): "Commute talk: How to get worse at programming"

Funnily enough, the tips given are somehow what everyone is encouraged to do on a daily basis.

1. "Always start with googling and finding existing solutions to your project / problem."

As if every project has to be created once only. That sounds stupid when you think about other fields. "Shit someone already made a bottle of water, I guess I should find another idea now."

If you want to be the first or only, doing that instantly kills your motivation. Work on the project because you personally enjoy it and want to be the person who can do it. Rule of thumb: A problem you really care about is one where someone else solving it still brings you joy. Mastery is not about competition.

The problem is that only "novelty" is rewarded (especially in research), pushing people to more hyped up topics without revisiting existing ones.

2. "Always check sites like StackOverflow to find good/best-practice algorithms to use for your problem."

Of course someone thought about the best data structure to tackle your problem, right? Why waste time thinking about how to combine what you know to get a grasp on using your knowledge to solve problems.

Remember, best practices (aka average practices) are a way for you to never practice.

3. "Look for libraries that already solve what you need to solve yourself."

Never re-invent the wheel, we have a deadline, why write so many lines and not build on other frameworks?

But if we use this library we benefit from free new features and fixes!

Valid points, but assess them every time and don't blindly go in that direction. Don't be the person who only puts libraries together. You can be the one creating your own, but that requires depending on yourself only.

4. "Never put your code on the internet."

Now that your baby is alive, never show it to anyone. What if someone says it's ugly?

Build this resilience step by step. I am guilty of not doing that myself, but this is healthy to have within the team. Healthy roasting sessions help in the right culture.

5. "When something just works, leave it and consider the project done."

That's the "what else can be done anyways? it works".

If you depend on external libraries and other people's best practices, you will feel that you have limited amount of control over your creation. This pushes you to avoid diving into nice rabbit holes because it's too much friction to change anything now. If you respect all the above, this becomes easier to avoid.

Just watch the talk, it's fun. I have so much respect for this man.

Sequence as Priority

Tue, 15 Jul 2025 00:00:00 GMT

import { Image } from 'astro:assets'; import topologicalSort from '../assets/sequence-as-priority/topological-sort.jpeg';

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Labeling 900 things as P0 is 100% guaranteed to drive dysfunction.<br/><br/>A human brain cannot handle the idea that a critical priority isn't being actively worked on or isn't already complete.</p>— Evan LaPointe (@evanlapointe) <a href="https://twitter.com/evanlapointe/status/1944795302755447261?ref_src=twsrc%5Etfw">July 14, 2025</a></blockquote>

Once you reflect a bit on your own experience, you realize that you need to switch to what Evan is mentioning as "sequence over priority".

I find myself doing that using dependency graphs:

Start by laying down all tasks as nodes. I use Obsidian Canvas for that. After listing "what should be done", now connect dependencies "what must be done to unlock another task". This should be hard dependencies. (simple e.g., API Design must be done before API Implementation.)

Once you have those, you need to topologically sort the graph you end up with. Here's a picture from the wikipedia page explaining topological sorting

You can see that many sequences are now possible. Each sequence ensures that if done one element at a time, all dependencies are respected. That means you would have 5, 7, 3 which could start in any order, and the rest can only appear in order of dependency.

At that point, it's about resource management and second look at this sequence. I personally aim for leading dominoes. Considering all else equal (effort required, urgency of project, etc..) 7 "unlocks more tasks downstream" hence shows as a nice contender to start with.

At this level it's a management decision on how to proceeed. That's what constitute your execution strategy. What is important, as Evan mentioned, is to be able to lay down a clear list. The hardest aspect here is maintaining/updating this task graph as you go. Especially when blockers happen, you depend on some team that suddenly can't deliver/is late, you had bad effort estimation and you need to shuffle things around, etc. (you will be punched in the face) Make sure to keep track of your failures and push yourself against the "everything is important because all customers are important" or anything similar. What's important is keeping your team going and having real impact. A million priorities are confusiing.

Practice so you can always answer "Now what?" and keep a healthy momentum.

TODO Streams

Sat, 07 Jun 2025 00:00:00 GMT

There is always more to do.

Every waking hour is one tempting you to tackle your never-ending todo list.
You know that by spending just 1hr now you would finish this or that.
"Finishing" your todo list feels so close, yet never achievable.

Aim for a todo stream instead.

A "list" feels finite and is handled as such.
You try to push through it to "be over, feel a sense of achievement, and relax".
A stream is seen as never-ending by nature.
Instead of finishing, you focus on handling the infinite stream without being overwhelmed.

You go from
"How can I finish everything I have to do?"
to
"What should I focus on today from this infinite stream?"

The stream being infinite is the norm, and you stop fighting it.

I first saw the idea in this "Rest in Motion" blog post referenced by Ben Kuhn in his amazing time management post.

What kind of Problem Solvers do you really need?

Tue, 03 Jun 2025 00:00:00 GMT

import { Image } from 'astro:assets'; import xyProblem from '../assets/problem-solving/XYProblem.jpeg';

Discover + Formulate usually feels like a burden and a lot of work for minimal short-term reward. The person trying to make an impact fast and maximize the use of their expertise will be irritated. Field Experts sometimes prefer focusing on the Solve, maybe with a touch of Formulate, but not the Discover. I personally believe you need to/must be involved in all of Discover, Formulate, and Solve stages regardless of your preferences. The person who knows how to Solve is best positioned to see the gaps in the Discover phase and scope the Formulate in such a way that Solve can be tackled the right way. This is especially true to avoid the XY Problem. At early stages in a startup or even team, you can't afford having someone who focuses only on one or the other part of this equation.

Fight the Glue Engineer Within

Tue, 22 Apr 2025 00:00:00 GMT

Glue engineers know the tools. They know the frameworks. They know which package to install. But ask them to build the tool itself — the database, the compiler, the scheduler — and they’re lost. They won’t build the next kernel. They won’t write the next compiler. They won’t create the next database. They won’t design the next OS. They won’t invent the next hypervisor. They won’t architect the next distributed system. They’re great at using the tools. But they’re not the ones who will build them. Glue Engineers usually work at an abtraction where they don’t have to think about the underlying systems. They can’t debug a kernel panic. They can’t write a compiler from scratch. They can’t design a new protocol. – Vipul Vaibhaw, How to Hire Engineers Who Ship Kernels

It's sadly so easy to end up working as a Glue Engineer.

When everything you're building only requires this level of "collage" skills to "bring shareholder value", you just find efficient ways to keep doing it.

The more you do it, the more you think it's the kind of Skills you need to become better. Because of course bringing more shareholder value increases your value, right?

Sometimes, and it depends which shareholders you want to make happy (career choice). You can keep solving similar problems your whole life and label yourself as a "React Software Engineer", as if carpenters would say "Red Hammer Carpenter", silly yet always used.

Do specialize, but don't limit yourself to tools you could learn to build yourself.

The best people I worked with are the ones who aren't afraid of building tools and have a VERY deep understanding of the fundamentals so they could build the blocks required to develop the solution.

This is also required to act as a "glue engineer" when required because how else would you assess the tools available?

I fight against slowly drifting to Glue Engineer every day, and put extra effort to push my team to think differently about how we tackle problems.

That's not easy, and it gets harder the less time you have for technical deep dives, but it's the only fight worth fighting.

Trust Your Gut in Hiring? Bake 10,000 Cheesecakes First

Sun, 29 Dec 2024 00:00:00 GMT

import { Image } from 'astro:assets'; import cheesecake0 from '../assets/cheesecake-hiring/cheesecake0.webp'; import cheesecake1 from '../assets/cheesecake-hiring/cheesecake1.webp'; import cheesecake2 from '../assets/cheesecake-hiring/cheesecake2.webp'; import cheesecake3 from '../assets/cheesecake-hiring/cheesecake3.webp';

The Perfect Recipe?

Every pastry chef can spot a perfect cheesecake at a glance: the smoothness of the surface, the slight jiggle, the warm glow at the edges. Not because they read recipe books but because they have crafted thousands. They have seen every possible variation, every mistake, every near-miss, and every success. Expert chefs also know that the true test of a cheesecake isn't in how it looks but in that first bite. What I realized is that hiring expertise follows the exact same path. We're all just learning to taste the cheesecake and choose the best one.

I chose the number 10,000 randomly as a big number, similar to the 10,000hrs rule.

</div>

How Hard Can It Be?

A failed cheesecake costs you ingredients and time. A failed hire can cost you your entire company. While anyone can follow a basic cheesecake recipe and produce an edible result, hiring requires a level of judgment that goes far beyond following a simple checklist.

Being at the core of what management does, it is a significant topic to get right.

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Management is mostly hiring well and firing quickly.</p>— Naval (@naval) <a href="https://twitter.com/naval/status/1536403651857510400?ref_src=twsrc%5Etfw">June 13, 2022</a></blockquote> The complexity of hiring manifests in multiple ways, as highlighted by many in the field

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Hiring is hard. I asked a bunch of founders who are crushing it at hiring what they believe contributes most to their hiring success. Five consistent themes emerged:<br />1. Captivating vision<br />2. A++ early team<br />3. Putting in the work<br />4. Founder's network<br />5. Pitching prowess<br /><br />Read on</p>— Lenny Rachitsky (@lennysan) <a href="https://twitter.com/lennysan/status/1412437666965032961?ref_src=twsrc%5Etfw">July 6, 2021</a></blockquote> <blockquote class="twitter-tweet"><p lang="en" dir="ltr">Almost all founder mistakes can be fixed. <br /><br />What can't? Hiring. <br /><br />And no, not just hiring the wrong person. <br /><br />It's not hiring enough, hiring too much, hiring the wrong type of role, or the right person at the wrong time. <br /><br />These are painful lessons. <br /><br />Learn them fast.</p>— Kristen Anderson (@FintechKristen) <a href="https://twitter.com/FintechKristen/status/1413622543961952256?ref_src=twsrc%5Etfw">July 9, 2021</a></blockquote> With stakes this high, how do we develop the judgment to make better hiring decisions?

The Three Ways to Judge: Make, Taste, or Trust

When it comes to assessing quality — whether in cheesecakes or candidates — we have three fundamental approaches we can follow. Consider yourself in front of the following 4 choices. Which one would you expect to taste better? Look carefully before answering.

<div style="display: flex; gap: 2%;"> <div style="flex: 1; text-align: center;"> <Image src={cheesecake0} alt="Cheesecake 1" style="max-width: 100%; height: auto;" /> <br /> <span>Cheesecake 1</span> </div> <div style="flex: 1; text-align: center;"> <Image src={cheesecake1} alt="Cheesecake 2" style="max-width: 100%; height: auto;" /> <br /> <span>Cheesecake 2</span> </div> <div style="flex: 1; text-align: center;"> <Image src={cheesecake2} alt="Cheesecake 3" style="max-width: 100%; height: auto;" /> <br /> <span>Cheesecake 3</span> </div> <div style="flex: 1; text-align: center;"> <Image src={cheesecake3} alt="Cheesecake 4" style="max-width: 100%; height: auto;" /> <br /> <span>Cheesecake 4</span> </div> </div> <br /> What are you even looking at when deciding? As a non-expert chef, maybe style, resemblance to some nice cheesecake you tasted once, or even fanciness. The features you look at are incomplete to make a reasonable final decision accordingly.

The Maker's Way: Mastery Through Practice

We all think we can spot talent when we see it, just like a good cheesecake. You definitely heard someone ask to write down more details about what they are looking for in a candidate, and their answer was, "I will know a good match when I see it." The difference between you and a chef is that you don't even know what to look at, let alone be able to assess it correctly once you see it.

The same goes for tasting it. You never trained your tastebuds, and after a specific point, everything tastes "really good." What do you do then? You choose the cheapest in this pool of "really good." This is one reason companies end up with slightly above-average teams. The quality you get is proportional to how well you trained your tastebuds.

So, do you need to be an expert? Well, kind of.

My theory is as follows:

The Cheesecake Theory: Authority in judgment is not given but earned through the deliberate baking and tasting of thousands of cheesecakes.

A Chef who has baked 10,000 cheesecakes has seen every possible variation, mistake, near-miss, and success. This Chef knows exactly what to look for in the shape and taste. Of course, that's only possible if the Chef deliberately practiced the process. The "deliberate" part is essential here.

You think deliberate practice has only marginal benefits?

Next to my place, I observe a homeless man playing the same accordion tune he's played for years. Despite likely crossing the 10,000-hour mark of practice, his melody remains unchanged, his skill unimproved. Not through any fault of his own - life's circumstances have denied him proper guidance and feedback. His daily performance serves as a powerful reminder: time invested without deliberate improvement is just time passed. When developing hiring expertise, let's ensure we don't fall into the same pattern of aimless repetition.

</div>

The Taster's Way: Expertise Through Evaluation

Is it the only way of assessing? Do I have to be a Chef? Not necessarily, but there is a reason why judges in all cooking shows are renowned Chefs (apart from the marketing aspect). Think about a Sommelier. They are the best at assessing the quality of wine but don't necessarily know all the tricks of the trade to make it. So, if you deliberately baked or even tasted a thousand cheesecakes, you earn the gut feeling that would help you assess one effectively.

Does it mean you can explicitly explain why a specific option is better every time? Not necessarily, and this happens with a lot of experts. Maybe you look at some code and say, "Well, this looks really nice," without being able to pinpoint precisely what makes something nice. Or maybe you look at chickens and can directly know their sex. I'm actually not joking; this blog post about Chicken Sexing and Perceptual Learning as a Path to Expertise explains this phenomenon really well. The TL;DR of this article is that perceptual expertise - like instantly judging code quality or spotting the sex of a chicken - can't be taught through instruction but is learned through immediate feedback over thousands of examples. Through this "perceptual exposure," our brains naturally detect patterns when shown many varied examples quickly, as proven when non-pilots matched experienced pilots' instrument-reading abilities after just two hours of targeted practice. This explains why true expertise in any field requires extensive exposure to high-quality examples with feedback, creating knowledge that experts can use but often can't explain - they just know what "feels right." It's also known as "developing taste." Yes, you have to earn the gut feeling, and you can actually train it.

Please do read the full article, it's amazing. One bad scenario I imagined is perceptual exposure with low-quality examples and how it could mess up your gut feeling, maybe permanently(?)

</div>

The Trust Way: Wisdom Through Proxies

Now, there is actually a third way other than deliberate baking and tasting. Consider the following: You are presented with two cheesecakes, one by your neighbor Nina and the second by Gordon Ramsey. Which one do you expect to taste better?

Without having them baked or even looking at them, you may say Gordon's because you know he's an expert. You know nothing about cheesecakes, how they are done, or how they should even taste. Yet, you were confident because someone somewhere gave the highest credit to that person. This is the third method which is assessment by proxy. You trust someone who graduated from MIT over a random local university in the Middle East because of their credibility, which is renowned worldwide. The same goes for referrals. If my super skilled friend referred you, you must be good. This only makes sense, although not always true.

What you didn't realize is that Nina, your neighbor, turns out to be Nina Métayer, who just became the best pastry chef in the world for 2024. Had you had an expert chef with you to help you decide, you would have quickly realized that her pastry is better. Now, you only know that because the title is there. Enjoy Gordon's "bad" cheesecake.

Relying on external assessment increases the probability, but you will still get "bad" candidates because you will be biased by others' assessments. More importantly, you will miss people who took a different route and can prove to be among the best. This is always a possible risk, no matter how you approach the hiring process. Considering limited time and energy, it is not humanly possible to be 100% fair with every candidate applying. At one point, you may have 2000 candidates for one position and need to filter it out as fast as possible. You will refer to some heuristics (rules of thumb) to do so.
Let's not fool ourselves into thinking we are always in such a situation and try to take shortcuts while we can have more extensive assessments.
Don't always listen to Bob from CheesyCake. His advice is based on the context of a big company; your context may be different. Take your time.

So the path forward is clear: become an expert, develop expert judgment, or consult experts. Simple, yet surprisingly controversial to many.

A common saying is "Game recognizes game," which points out how experts have the gut feeling to recognize other experts or even hidden potential.

Mastering the Craft

If you have read this far, my words have made some sense to you, which is a good sign. Now what?

We mentioned choosing the cheesecake that would taste best, but what does that actually mean? "Best" is very subjective and should be treated as such. What makes a good candidate at one company may not make them good at another, even for the same role.

The important part is not fighting about a universal definition of "best," but being aware of a clear definition of your version of "best." This may seem like a trivial step, which is partly why it is usually overlooked with assumptions like "I will know it when I see it." Well, maybe, but you won't attract it if you don't clearly advertise what you want, which means you may never actually get to see it.

About the Candidate

What makes a great candidate is what makes a great employee; they should

Tackle the right tasks (Effective)
Tackle tasks the right way (Efficient)
Deliver high-quality outcomes (Expert)
Learn and grow fast (Eager)

You may require different levels for each or even add more aspects you believe are important for you, but the above list shows the essential core traits I believe are necessary.

Some argue that Eager is not always required. Would you rather choose the Chef who has consistently made a 3/5-star cheesecake for the past 20 years or a new chef who has only been self-taught for a few months, showing a 2/5-star cheesecake?
If all you ever care about is a 3/5 quality, your choice may be clear.
If you aim for a 5/5, this first Chef is a clear red flag.
You choose how to balance those dimensions based on your needs (yes, it is essential to have them clearly defined).

It goes without saying that not being an asshole is on top of the list. Sadly, too many people still overlook this aspect and consider that Expertise would cover up for it. There is a reason why "One rotten apple spoils the barrel" is such a famous proverb. Let's add one line to reflect that

Fit well in different environments (Adaptable)

</div>

About the Person hiring

While you could focus on everything during the assessment, at the end of the day, a human is doing the assessment, and humans are biased—not only biased but emotional and sometimes irrational.

As professional as you may be, the implicit clues you pick up can sometimes be based on your own personality, how attractive the person is, or how clear their voice is. The candidate may even remind you of someone you hate or love. Both positive and negative biases negatively affect an effective assessment.

Some tips

First, read books about decision-making so you can recognize your own biases. "Thinking in Bets" by Annie Duke is a great start.
Write down evaluation criteria before meeting with the candidate. Knowing what you want, what you need, and how to clearly state that out are three different challenges at this stage.
Document decisions immediately after each interview, not later. You may believe that you know an implicit definition of "good," but challenge it by writing it out or explaining it out loud to someone to convince them. For instance, in many situations, you may find that what you consider good is affected by the recency bias. e.g., today's good candidate seems to you better than the other good candidate from 3 weeks ago. (no notes / clear metrics? you fell into the trap)
Make sure your filtering process takes into consideration the context of the candidates. Consider two candidates: one from a prestigious university, good grades, with no work experience, versus one from a good university who worked as a waiter throughout their studies to support their family. Which path tells you more about their potential? Take some time to reflect on your own hiring biases.

Questions to ask yourself

If the candidate was of a different age/gender/background, would I feel the same way? Try to replace them with someone you know and like or someone you specifically don't.
Can I point to specific examples that support each of my judgments?
What makes me say they're a 'culture fit' - am I confusing comfort/easygoing with competence?
Which of my past hires succeeded/failed in unexpected ways, and what did I miss initially?
Where have my instincts been right/wrong before?
Have I given equal weight to similar strengths/weaknesses across all candidates?
Am I rushing to judgment because I'm under pressure to hire?

Please don't read books before ordering a cheesecake, but at least do so before making decisions that can affect the lives and careers of candidates and your team/company when hiring.

</div>

Remember: Just like developing any expertise, reducing bias requires both structured systems and self-awareness of our judgment patterns. It's not about eliminating intuition but about calibrating it properly.

Personal Rules of Thumb

"Theory without practice is empty, and practice without theory is blind." - Kurt Lewin

No matter how much you read about hiring, you have to experience it firsthand and fail badly to learn.
Suffering is a required part of the process.
You can learn from the experience of your colleagues hiring in other teams. Learning from others' mistakes is required when your samples are small over your career.
Allow your process to detect exceptional talent. Your challenge should be hard enough for excellent people to pass it but still allow exceptional geniuses to stand out. An easy challenge would reach a "saturated good solution," which is marginally different between candidates with dramatically different skill sets.
Humans are not rational by default; if you think you are, read "Predictably Irrational" by Dan Ariely.
No matter how much you read about decision-making and decision biases, you will still make mistakes. Make sure to actively try to be aware of them and calibrate accordingly.
Great recruiters are worth their weight in gold. They have seen a lot of cheesecakes and a lot of Chefs in different teams, contexts, and industries. Their gut feeling is worth considering.
Not every great employee can hire well, but most employees who hire well are great.
Always reflect on your skills at assessing people and outcomes. You may be good at one and bad at the other.

Summary

The Cheesecake Theory: Authority in judgment is not given but earned through the deliberate baking and tasting of thousands of cheesecakes.

Your gut feeling is the dark matter of hiring. With time, it starts to make up 95% of our decisions. Make sure you train it with high-quality examples.

Your first cheesecake won't be perfect. Neither will your first hire. But with each attempt, your judgment grows sharper. Start counting your cheesecakes.

What's your approach to developing hiring expertise? How many "cheesecakes" have you baked so far? Interested in your experience and feedback, let's connect on Twitter/X @jeanmarcalkazzi.

Let's Talk Time Management

Sun, 02 Apr 2023 00:00:00 GMT

import { Image } from 'astro:assets'; import xkcd from '../assets/time-management/xkcd_time_management.png'; import schedule00 from '../assets/time-management/schedule_00.png'; import schedule01 from '../assets/time-management/schedule_01.png'; import tasktable from '../assets/time-management/tasktable.jpg'; import mindForNumbers from '../assets/time-management/A-Mind-for-Numbers-Learning-Strategies.webp'; import cornellNotes from '../assets/time-management/CornellNotes-Notepad2d-1024x1018.jpg'; import plannerInstructions from '../assets/time-management/planner_instructions.png'; import plannerFilled from '../assets/time-management/planner_filled.png'; import things3 from '../assets/time-management/things3.png'; import obsidianCanvas from '../assets/time-management/demo-help-vault.png';

Intro

First, what is time management?

According to Wikipedia

Time management is the process of planning and exercising conscious control of time spent on specific activities, especially to increase effectiveness, efficiency, and productivity.

Well, fair enough. As long as we don't end up like this by the end of this post, we should be fine.

Now how did I end up able to juggle a full-time job at a startup, an ongoing PhD, staying fit, keeping a social life, and staying (arguably) sane in the process?

If you are at school/university, I believe the story steps will be insightful. If you don't feel like knowing the story behind how I got to my current approach, just jump to the end directly (no hard feelings).

</div>

Mom: Where are you? When are you coming home?

For the first 2 years of university, I was working part-time giving private courses in Math and Physics for school students to be able to afford my studies. I used to only take students who are of walking distance to my place, on the same days of the week. So my whereabouts were somehow clear. At one point, life decided to make things a bit harder, and I needed to find a bigger source of income, which led me to taking a job as Lead Data Scientist at a startup as well as leading projects in our university research Lab. Things got way more complicated very fast and I needed a way to keep up. Not only that, but having a random university schedule and 2 jobs made me jump around between multiple locations, and made my mom confused on what food to prepare for the day and when to expect me to be home for dinner.

Food was a main concern (I mean, it's mom), but it is worth noting that the situation in Lebanon was bad at the time, and random explosions used to happen everywhere throughout the country. So keeping my mom up to date on my location was necessary to make her feel better. All she cared about is knowing that I am safe, and have enough food to survive the day.

</div>

To avoid answering multiple calls a day with the sole reason of knowing where I am physically, I created a simple schedule of the week showing where I am suposed to be at any point in time and hung it in the living room. The deal was: the schedule is true unless I inform you otherwise.

One day of the week looked something like this:

This turned out to be an amazing solution.

Safety
Food

Until we started talking about what I am doing at University, and how the courses are going. Mom wanted to know what is happening in general, Dad liked to go into more details on what I was learning. The more complex the courses were, the harder it was to keep track of their names.

Dad: What is the name of this course again?...

I enjoyed our discussions a lot, and wanted us to stay in sync. This was an easy fix here, right? Just add the course names instead of "University", and they will be there on the wall as a reminder.

Which looked something like this:

Now we have

Safety
Food
Synced Discussions

This is nice, but.... we still didn't solve the fact that I am working on a gazillion things at once and I need to keep my head straight.

The Enlightment

As I was doing this last detailed schedule, I realized that I was more often than not checking it to know what I should be doing next, and which course to attend. I slowly started adding the course location (building+room), and accounting more accurately for the commute times, while adding some "Relax" slots in the process.

There was also this nice side-effect of feeling less overwhelmed because I know for a fact that a some point in the day I will tackle problem X for 3 hours, so I can offload overthinking about problem X until I reach this time-slot.

Schedule Management

At this point, and maybe through some searches about time management, I landed on a blog post which was freshly posted: "The Importance of Deep Work & The 30-Hour Method for Learning a New Skill" by Azeria.

Which I not only highly recommend you to read, but would even give you a cookie if you do.

This was the first time I heard about "Deep Work", which convinced me to try it out as I had a similar experience which randomly ended up like a time-blocking strategy.

This is when I got the Cal Newport's Book Deep Work, and dove into it.

Reading the book, one of the main ideas was about Time-Blocking, which looked something like this:

Looks familiar, right?

I only reached such a point from personal experience, but Cal Newport was explaining scientifically why this is a great approach to plan your day and it was beautiful to read.

The main idea explained simply is: The fact of planning your day in blocks of time will help you

Stay more focused on the current task at hand.
Improve your effort estimation on tasks.
Have clear expectations from yourself by end of day.
Be more efficient overall but still adapting to different things life throws at you throughout the day.

Get the book, and block 30min a day to read it. You will thank me later.

Maximizing Efficiency

Now that I know when I am working and on what, I needed to maximize my efficiency in each one of them, namely:

Research Lab Work
University Courses (Class + Lab + Exams)
Startup

All of which had 1 thing in common: Robust and Fast Learning.

Learning

To learn this skill, I decided to take the "Learning how to Learn" course offered by Barbara Oakley and Terrence Sejnowski.

The 15hours course was great, but I craved more details. So I got the book on which this course was based, "A Mind for Numbers".

This book helped me understand how learning and memorization work (from a neuroscientist' perspective) so I could leverage the approaches proposed in a practical way.

Simply put, here are some of the main points mentioned.

Do you know why starting with the hardest problem in your exam is actually more beneficial than starting with the easy one? You'll find the answer in the book, and I used this trick a lot.

Note-Taking

To learn something effectively, you need to have it well represented for efficient learning. This means you need to take good notes in class. After searching for a while, I ended up with the Cornell Note Taking System.

You simply split the page in 3 sections: main notes, cue questions, and summary.

During class, you write in the main notes section.
You then re-read it while revising and summarize the main notes in the summary section.
While reviewing, you write down important details/questions in the cue questions section.

The Cornell note-taking system is based on research that shows that actively engaging with information improves retention and recall of information. By dividing information into different sections, the Cornell method enables you to process information at different levels, leading to better comprehension and memory retention. Additionally, the summary section provides a quick review of the main points, making it easier to recall information when needed.

This ended up complimenting a lot of the ideas mentioned in "A Mind for Numbers".

My Current Approach

Fast Forward few years...

<blockquote class="twitter-tweet" data-theme="dark"><p lang="en" dir="ltr">I may be taking <a href="https://twitter.com/ProfCalNewport?ref_src=twsrc%5Etfw">@ProfCalNewport</a> 's Deep Work approach too seriously<br /><br />Been doing so since ~2018, just recently started using his Time-Block Planner.<br /><br />Actually it's been amazing.<br />This peace of mind of knowing that "I can focus only on X now because Y is planned for later today". <a href="https://t.co/Erl9bnTn9C">pic.twitter.com/Erl9bnTn9C</a></p>— Jean-Marc (@jeanmarcalkazzi) <a href="https://twitter.com/jeanmarcalkazzi/status/1612198960260694018?ref_src=twsrc%5Etfw">January 8, 2023</a></blockquote> Now, how does it work practically?

There are multiple aspects involved, of which I will cover the following:

Schedule Management
Task Management
Note-Taking

Schedule Management

The tweet is pretty clear I believe, Deep Work all the way. The empty planner looks as follows. Each day consists of 2 pages: one for tasks, ideas and metrics, the other for the detailed schedule.

The Work Week

After experimenting for a while with such an approach, I ended up with a version that fits my own schedule, which you can see here.

On the left side, I nearly followed the same approach mentioned in Cal Newport's book, while color coding the "Important Ideas" in red. This helps me whenever I am doing my weekly retrospective.

On the right side, you can see my schedule crosses till night, including personal aspect (outside of my daily job) which I saw to be convenient to see in one place. Red here shows blockers I should not try to change on the same day (either involves other people in a meeting, or involves food [very important]). All other blockers are 'equal' in a sense. I try to add a "Slack Time" blocker, maybe more on other days, because regardless how well you plan, you need to adapt when faced with the real world. This is where I cross out blocks I thought I would do, when something urgent comes up.

You can't have a company incident you need to react to and say "well, I had something planned at this time, maybe come back tomorrow?".

</div>

The idea of crossing out instead of erasing is very important. Once you do the retrospective, you need to know where you miscalculated in your planning and adjust accordingly for the week after!

Try to always add in a margin for errors. The worst that could happen is finding yourself trying to keep up with the time blocks which start leaking into the next ones and you end your day frustrated. Priorities changed throughout the day? Adapt accordingly. The goal of the planner is not to be written in stone, it's about giving you visibility of what you are set out to achieve today while helping you improve your planning skills and keep track of what you are achieving.

The Weekend

So, do you do the same on a weekend? The answer is, yes and no. I use Saturdays as a "no screen, no work, just relax" days. This is also a day I try to spend outside with my girlfriend, exploring new places and activities.

Sundays are "structuring, planning, learning" days. This helps me keep up with house chores, missed tasks, and learning new things to feed my curiosity and be ready to tackle harder and harder challenges.

Take breaks as much as you need. No amount of work or success is worth sacrificing your mental health, no matter how much you try to convince yourself that it makes sense.

</div>

Task Management

Do I plan tasks only by writing them down on the planner? But what about tasks that should be done "someday" but maybe not this week? How to keep track of a plethora of things to do?

I use Things.

You have your:

"Inbox", where you add tasks before deciding when/what to do with them.
"Today", where you can have all tasks due today.
"Upcoming", where you can see future events (if calendar sync on), and tasks.
"Anytime", do it anytime you want?
"Someday", do it someday, maybe?
"Logbook", where you can see the history of completed tasks.

I also use what Things calls "Area" and "Project", where I have 2 areas "Personal" and "Work". In each I have different projects, each of which can have its own tasks. This turned out to be very similar to the PARA Method explained by Tiago Forte in his book "Building a Second Brain" which I recently read and helped me fine-tune my process (I use a similar approach for Note-Taking on Notion and for ToDos on Things3).

I like this structure for multiple reasons, but 1 in particular: I can keep track of the balance of number of tasks spread out across projects, and I can literally "see" procrastination, which triggers me to act on it.

You will see more about the steps of creating and planning ToDos in the Workflow section.

Note-Taking

Now that I am no longer taking classes at university, the Cornell Note-Taking System does not really fit my needs. This is where I switched to a somehow custom Bullet Journal approach which I had started during University when it was just a way to track what I was doing during my work at the startup and the univeristy's research Lab. This is now the "Ideas" section of my planner, where it is more of a freeform section, open for thoughts and reflections.

Notes written and not retrieved are slightly better than notes never written, but not really much more useful. That's why I have a private GitHub repository where I copy all my notes each week, and try to augment them with some insights from having a global picture over what happened. The interesting bit about the planner being only 13 weeks long gives you a bounded scope for covering only 1 quarter, helping you do a retrospective once done, and gives you the opportunity to "strart fresh" with the next quarter, helping you stay more and more consistent.

I also recently started using Obsidian for brainstorming with their new Canvas feature. I love global visual overview of all notes and how they relate to each other, and it turned out to be very helpful while writing my thesis now. Maybe worth another blog post for a deep dive in this one? We'll see.

The Workflow

Whatever you read next, the most important aspect is consistency.

No amount of "10 things to boost your productivity" blogs will improve your life if you don't stay consistent in the execution of your plan and flexible in its creation. This should have its own blog post, but for interested readers, you can check out this book.

</div>

Let's go through a scenario to get a glimpse of the workflow. It's Sunday, you read this blog post, and decided to act on it.

Write the top 3-5 big goals to achieve over the week.
For each of the goals, split it into relevant tasks spread out through the week. You can skip to day X and write the relevant ToDos before reaching the day.
Draw Monday's schedule in details, based on your hobbies, work schedule, and personal events. Add relevant tasks to complete the weekly goals.
Each day, at night, go through the tasks of the day, re-schedule what is not done, and plan the next day by drawing the schedule in details, including the tasks from your ToDo app. Copy the daily notes to your digital app of choice (in my case a GitHub repo, everything written in Markdown).
Next Sunday, do your weekly goals' retrospective, as well as writing notes of what you learned from the previous week.
Repeat steps 1 to 5 for the next week, incorportating what you learned about the dynamics of your days and what works for you.

More planning usually happens at the start of each year + each quarter, as well as needed retrospectives, but they usually follow similar approaches to how weeks are planned, just on a bigger scope.

You want to improve? You need to understand your current level to realize how you can be pushed to/above your limits. This is what my workflow is about, and how I keep refining it.

Final Thoughts

Did I add every single detail? Probably missed some of which I do very intuitively now. The goal of this post is not only the resulting workflow, but the path taken to reach the final result.

The whole point of showing the path from where I was to where I ended up is to prove once more that you can figure things out. My 2 golden questions in life: How hard can it be? What could possibly go wrong? push me to take action sooner rather than later.

</div>

You don't need to wait for the perfect timing, book, course, or mentor to teach you anything. Just start tackling your problems with what you have now. Whatever new insights you can learn along the way will enrich your current approach.

Is this the final best possible version of planning my life? Surely not. Does every week look the same? Also not. Should I feel depressed knowing that I will never reach a perfect strategy? This assumes there is a perfect strategy. By the time you read it, I may have modified my approach. The point is to adapt and focus on improving, never reaching a stale state.

Different people will have different preferences on how to manage their time and life. There is no one-solution-fit-all approach. Take away what you need from every reference you consume and build your own personalized approach.

Understand Python Environment management through a hiking trip.

Mon, 20 Feb 2023 00:00:00 GMT

import { Image } from 'astro:assets'; import theImportantField from '../assets/python-env/the_important_field.png'; import textual from '../assets/python-env/textual.png'; import sandwich from '../assets/python-env/sandwich.png'; import multiplePythonVersions from '../assets/python-env/multiple-python-versions.png'; import pythonEnvironment from '../assets/python-env/python_environment_2x.png'; import realProgrammers from '../assets/python-env/real_programmers.png';

Let's learn about Python Environments as an adventure, shall we?

You can find a lightning talk version of this blog post given at PyData London 2022 here!

</div>

Introduction

You like going on adventures.

An Adventurer's Life

For every adventure you need different kinds of tools. Every trip, it takes you hours to correctly prepare your backpack. Later on you decide to buy one backpack per type of adventure. A backpack for hikes, another one for rock climbing, and a third one for canoeing. You soon find out how easy it is to only care about planning the adventure itself when your tools are ready. One day you find yourself interested in kayaking, camping, mountaineering, and skydiving, which made managing too many backpacks a hassle. You then decide to hire someone to take care of preparing your backpacks and tools for each trip. You tell them what tools you need for your trip, they prepare the backpack and tools needed, and you care about planning your adventure. Some special tools can be found in (few) places, so you try to specify where to buy them from (we don't want our assistant to search all possible shops!). This sounds more manageable, scalable with your neverending interests, and easily maintainable. What a beautiful relaxing adventurer life!

You may say

Wait, what on Earth does is have to do with Python Environments?

A Programmer's Life

Well, Python programmers (and programmers in general) face very similar problems. They start by (maybe) being interested in Statistics, Graphical User Interfaces (GUI), and some REST APIs. Every one of which needs different libraries [tools] to get the job done. Installing [packing] all those libraries [tools] on your system and carrying all of them around for every project [adventure] could be confusing. What if this Statistics project only works with the old awesome-lib version 2, but the new REST APIs project needs the newer awesome-lib version 3? What if I need to run my Statistics project somewhere else, do I install all libraries [tools] I ever used for all of my projects? If not, how can I find out which tools are for which project?

Luckily there's a solution.

You may ask

Can we just hire someone to prepare my "backpack" for every project?

The answer is yes! You can hire conda, pipenv, poetry, virtualenv and that's their job! Best part? They do it for free and you can specify different locations to get your special tools (libraries) from.

You can even hire someone like pyenv to buy the backpacks you need.

Those are possible to be done for free thanks to amazing communities and open source contributors spending time and money to make it happen. Always consider contributing back to the community with your time, money, helping others, or any way that fits you.

</div>

The Analogy

Adventurer	Python Programmer
Trip	Project
Backpack	Python Environment
Backpack Type	Python Version
Tools	Python Libraries
Assistant	Python Env and Package Manager
Tools Shop	Repositories

Now how does it work in practice? Do you need to learn about all of them? What do they have in common? How can they pack my backpack? How to use and manage this backpack?

This blog post will walk you through how to prepare and manage your backpack for a small dummy project.

How hard can it be?

Let's get back to our Analogy

Adventurer	Python Programmer
Have a Trip goal	Have a Project type
Prepare a new empty backpack	Create a new empty Python Environment
Decide on the tools needed for the specified trip	Decide on the libraries needed for this project
List down the shops from which the tools will be bought	List down repositories to download libraries from
Hire an assistant for the job	Install a Python Environment and Package Manager

The One Backpack Approach

If you were to use one backpack for all your trips, the only thing you need to care about is preparing your tools.

Let's start with this scenario, and then move on to learning about preparing multiples backpacks.

In Python, we have a standard way of installing packages (libraries) using a tool called pip. pip is a recursive acronym and stands for pip installs packages. Sometimes pip does not come with Python, so we have to install it ourselves.

Whenever you need to install anything, check if you have it first. After installation, check if it was correctly installed. Developers, more often than not, skip the 'checking' phase and only follow tutorials blindly then spend their time debugging bugs they could have avoided by simple sanity checks!

</div>

Just make sure to double check the correct things...

Finding Python

On Mac and Linux, you usually have two Python distributions already installed on your system: Python 2 and Python 3 (Although Python2 reached end of life and won't ship with newer OS versions anymore). Those specific Python distributions are used by internal components in the Operating System (macOS and Ubuntu), which makes them less favorable to be messed up with for other projects. Let us leave them in peace, and use our own different Python distributions for development, so we don't mess up anything being used by our system.

You can check your current versions of Python2 and Python3

python --version

Output

Python 2.7.17

By default, python is linked to Python 2 on your system. For Python 3, you will use very similar commands to Python 2, but adding the 3 to it: python3. Newer systems however have python linked to Python 3, so just make sure to check.

</div>

python3 --version

Output

Python 3.6.9

What if we install our own Python 3.6.9, how can I know if I am using the default one on the system or my own?

which python3.6

Will return the path to the binary executed when running python3.6, in our case:

/usr/bin/python3.6

You can safely consider that anything under /usr/bin is used by your system by default and should not be messed up with.

</div>

For the rest of this tutorial, we will be using Python 3.11. Python 2 reached end of life on January 1st 2020.

If we don't want to use the system python, how can we install our own?

A New Backpack

There are several ways to install a new Python version alongside the others:

Install it directly on the system.
Use pyenv to manage python installations on your system.
Use conda to install relevant python version on environment creation.

If we go for Option 1, we can technically install any Python version either from pre-built binaries or from source. Either ways will end up with the system having multiple python versions installed with no single point to manage all of them.

If we go for Option 2, pyenv will manage multiple Python Versions on your system and save you the hassle of the manual work.

If we go for Option 3, we will hand everything to conda from python version management to environment management.

Option 2 looks like Option 1 but without the hassle. Option 3 looks easier than Option 2 and will be left as an exercise to the reader.

Using PyEnv

Quoting the authors of the library:

pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

We said that, by default on Ubuntu 18.04, we had Python 2.7.7 and Python 3.6.9 installed. We also said that we will proceed with Python 3.11 for Textual. How can we download/setup/install/whatever has to happen to have Python3.11 ready using pyenv?

Please follow the Installation Instructions before proceeding.

</div>

PyEnv Concepts

Global Python Version: The default version we want our system to be using at all times.
Local Python Version: A specific Python version we want to be used when we invoke python in a specific directory.
Python Installation: A specific python version installed in the root directory of pyenv

We need Python3.11, so we will install it on our system using pyenv

pyenv install 3.11

This will install the latest available version of 3.11:

python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.11.1.tar.xz...
-> https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz
Installing Python-3.11.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.11.1 to /Users/jeanmarcalkazzi/.pyenv/versions/3.11.1

This will download and install Python-3.11.1 on the system. To make sure it worked:

pyenv versions

* system
  3.6.9
  3.9.4
  3.11.1

We see that 3.11.1 is listed in the available versions. Being listed is one thing, and being usable is another.

As far as Pyenv is concerned, version names are simply directories under $(pyenv root)/versions.

Which means that our python executable should be under $(pyenv root)/versions and will invoke the Python Interpreter.

Input

$(pyenv root)/versions/3.11.1/bin/python

Output

Python 3.11.1 (main, Feb 20 2023, 22:23:32) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Well, I guess it works")
Well, I guess it works
>>>

Now that we have our backpack, we need to get the tools.

Let's try to install numpy

python3.11 -m pip install numpy

Why use python3.11 -m pip install X instead of just pip install X? First option will ensure that we are putting the right tool in the right backpack, in our case, 3.11. For more details check out this great post by Brett Cannon who is a Python Core Developer and amazing writer!

</div>

Now we have numpy-1.24.2 installed in the pyenv's python3.11.1 installation. This is great, until you have another project which also uses python3.11.1 but needs an older version of numpy. How can you achieve that? Having only one backpack won't help, and given that the backpack is virtual here and we can have as many copy as we need, why not use it as a blueprint and just clone it for each new project with such a needed version?

The Multi-Backpack Approach

Our goal is to manage different Python environments for different projects. A Python environment is just a separate Python backpack on your system where you will install any library you need. To do so, we will:

Hire a Python Environment Manager
Ask it to create a new environment
Tell it about the libraries we will need to use in this environment

Once our backpack (environment) and tools (libraries) are ready for our next adventure, we will start using them.

So what we need is:

A Python Environment Manager to use.
Creating a new Python Environment (Backpack) based on the versions we are managing (with tools like pyenv).
Have a list of tools (libraries) to prepare for the adventure.

One possible env manager is poetry. You may find a lot more, and by the time you read this post you could have multiple other tools released, but just focus on what you need to be done, then choose a tool which can do it for you.

The Trip Goal

We want to create a small project to test out an amazing new library called Textual.

I mean, just look at what it can do!

Preparing a new Backpack

At the time of writing, the below commands are up to date. Those could change and things could break, so take away the essence of what needs to be done, and execute it given the new version of the tools you choose!

</div>

To prepare your new empty Python Environment, you need to know which Python version is required by your library.

Checking Textual's GitHub Repo, we find

Textual runs on Linux, macOS, and Windows. Textual requires Python 3.7 or above.

We will go for Python 3.11 in this case. Given this Python version, what to do with it?

We previously mentioned:

A Python Environment Manager to use.

So we will follow the docs to install it first.

Running

poetry --version
Poetry (version 1.3.2)

Shows that the installation was successful.

Now we create a new "backpack" by running

poetry new textual-demo

Switching to the newly created directly, we find the following structure

.
├── README.md
├── pyproject.toml
├── tests
│   └── __init__.py
└── textual_demo
    └── __init__.py

3 directories, 4 files

We also mentioned

Creating a new Python Environment (Backpack) based on the versions we are managing (with tools like pyenv).

To ensure that the python version you need is used, run the following:

poetry env use $(pyenv root)/versions/3.11.1/bin/python

Using virtualenv: /Users/jeanmarcalkazzi/Library/Caches/pypoetry/virtualenvs/textual-demo-WRrLXqmL-py3.11

This will make sure that the version installed by pyenv previously is cloned into its own directory for this project.

You can check that by opening the pyproject.toml file and checking

python="^3.11.1"

We still have the last point in our list

Have a list of tools (libraries) to prepare for the adventure.

This is the important one, deciding on what will be installed specifically for this project's python version which was cloned into its own directory.

Given that we need textual, we run

poetry add textual

Using version ^0.11.1 for textual

Updating dependencies
Resolving dependencies... (2.3s)

Writing lock file

Package operations: 25 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing markupsafe (2.1.2)
  • Installing mdurl (0.1.2)
  • Installing python-dateutil (2.8.2)
  • Installing pyyaml (6.0)
  • Installing click (8.1.3)
  • Installing ghp-import (2.1.0)
  • Installing jinja2 (3.1.2)
  • Installing markdown (3.3.7)
  • Installing markdown-it-py (2.1.0)
  • Installing mergedeep (1.3.4)
  • Installing packaging (23.0)
  • Installing pyyaml-env-tag (0.1)
  • Installing uc-micro-py (1.0.1)
  • Installing watchdog (2.2.1)
  • Installing linkify-it-py (1.0.3)
  • Installing mdit-py-plugins (0.3.4)
  • Installing mkdocs (1.4.2)
  • Installing pygments (2.14.0)
  • Installing zipp (3.14.0)
  • Installing importlib-metadata (4.13.0)
  • Installing mkdocs-exclude (1.0.2)
  • Installing rich (13.3.1)
  • Installing typing-extensions (4.5.0)
  • Installing textual (0.11.1)

After that, you should be able to use textual in your own isolated environment, with a pyproject.toml file describing the environment, making it reproducible and avoiding arguments like "It works on my machine".

Looking at the pyproject.toml file, we can see the following

[tool.poetry]
name = "textual-demo"
version = "0.1.0"
description = ""
authors = ["Jean-Marc Alkazzi <...>"]
readme = "README.md"
packages = [{include = "textual_demo"}]

[tool.poetry.dependencies]
python = "^3.11.1"
textual = "^0.11.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Whenever you are faced with an adventure now, you know how to choose your backpack, how to buy your backpacks, how to manage them, how to clone them for each adventure, and how to get the tools you need without messing up the rest of your adventures!

</div>

What could possibly go wrong?

sudo is short for super user do. Which tells your system to do something with elevated priviledges. Don't use this super power on everything by forcing your system to execute every command this way. The only time you would need such power is to install system-related packages. If you find yourself using sudo to force every command to just run, you should probably take a deep breath.

If you mess up your Python installations, you could get too many python versions on your system

Hiring multiple assistants to take care of all your backpacks, and ending up with 8 different "hiking" backpacks with different tools. Now this will be a confusing situation.

This xkcd post shows what we want to try with the above approach.

One more tip, just make sure you don't go into the vim/emacs battle and end up using butterflies

Just use the tools you like, as long as you are fulfilling the required behaviors.

Until next adventure!

Setting up your NVIDIA environment without suffering.

Sat, 19 Sep 2020 00:00:00 GMT

import { Image } from 'astro:assets'; import unluckyBrian from '../assets/nvidia-env-setup/unluckybrian-learn-pytorch.png'; import cudaToolkit from '../assets/nvidia-env-setup/cuda-10.2-toolkit.png'; import cudaChoice from '../assets/nvidia-env-setup/cuda-toolkit-choice.png'; import nvidiaSmi from '../assets/nvidia-env-setup/nvidia-smi-102.png'; import nvsciError from '../assets/nvidia-env-setup/nvsci-error.png'; import cudnn803 from '../assets/nvidia-env-setup/cudnn803.png'; import tensorrt7Download from '../assets/nvidia-env-setup/tensorrt7-download.png'; import tensorrtInstallCheck from '../assets/nvidia-env-setup/tensorrt-install-check.png';

Updated on 02.09.2022 due to a change of NVIDIA's public key.

</div>

Updated on 09.03.2021 to include warning about 'Secure Boot'.

</div>

Updated on 02.02.2021 for ISAAC 2020.2

</div>

Your NVIDIA Environment

Let's say you just heard about NVIDIA Isaac and you are excited to test it out. You went to the setup documentation first and found:

To avoid potential issues caused by mismatching library versions, we recommend using CUDA 10.2, CUDNN 8.0.3, and TensorRT 7.1.3, which are the versions available in Jetpack.

This is a very important step to take whenever you need to use any tool: check the recommended environment setup. Do not assume that the latest version of everything is the go-to solution.

Now we know that we need:

CUDA 10.2
cudNN 8.0.3
TensorRT 7.1.3

This is for Isaac, but even for Deep Learning you would need to have CUDA, cudNN, and TensorRT. We will explain why those tools are needed in this post.

The setup is done on an MSI GS66 Stealth with an NVIDIA RTX 2070.

The updated setup (ISAAC 2020.2) uses an MSI GS66 with an NVIDIA RTX 2080.

Prerequisites

Ubuntu 18.04
A good internet connection (We will download ~5GB)
Patience

At some point in time, we were all like Brian.

This blog post will ease out the burden of setting up your environment and show you clear steps to take. Let's dive directly into it.

Before going into copy pasting commands and hoping for things to work out, we need to understand why we are installing those "tools" on our system.

</div>

Before proceeding, you may have to disable 'Secure Boot' in your BIOS to avoid issues with installed drivers.

</div>

CUDA

What is CUDA

"CUDA is a parallel computing platform and programming model that makes using a GPU for general purpose computing simple and elegant." -NVIDIA Blog

What does that even mean?

It means that having CUDA on your PC will allow you to use the CUDA extensions and libraries (which is part of the CUDA toolkit) for the programming language you are using (ex: C++), to specify which parts of your code should be parallelized and how. This parallelization is possible thanks to the GPU's architecture. You can check out this fun experiment by Mythbusters painting the Mona Lisa in milliseconds in a "GPU way".

If, in our case, we need CUDA to take advantage of this parallelization on the GPU, why not any other parallel computing platform? CUDA is the official platform provided by NVIDIA for NVIDIA GPUs and supports other low-level APIs like OpenCL.

Installation

In our case we need CUDA version 10.2, but what if later on we want to use this same tutorial to install CUDA 11?

You should be able to find out where to get CUDA from instead of copying commands which will work for the current needed versions.

This blog will show you where to look for the tools we will be installing.

You can find all CUDA releases by checking the CUDA Toolkit Archive In our case, we click on the link for CUDA 10.2 (November 2019).

We are redirected to a page where we can choose the OS, our PC's architecture, the OS distribution, OS version, and installer type.

We will choose Linux, x86_64, Ubuntu, 18.04, deb(network) as shown in this image.

deb means debian package. Ubuntu is built on top of Debian GNU/Linux. A debian package is the installer file used by Debian Linux and all its derivatives (Including Ubuntu).

(network) means that we will let the package download all additional data through the network (internet connection). This explains why the file is ~3KB only.

Run the following commands to download relevant files and start the installation

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin

sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub

sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"

sudo apt-get update

sudo apt-get install cuda-10.2

Notice that the last command includes the version we want to install 10.2. If we were to install version 11.0 we would download the relevant files and on this last step install cuda-11.0. Choosing cuda-10.2 will download both the cuda-toolkit and the relevant drivers.

To point your system to where cuda is installed, we will add the relevant environment variables to the end of our .bashrc file.

export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Just as a good practice, reboot your PC after installing/upgrading new drivers.

</div>

Verify Installation

After rebooting, we need to check that CUDA 10.2 was actually installed. This is done by running nvcc -V

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

We see that CUDA release 10.2 is installed, with an exact version V10.2.89

Having the installed packages/drivers is one thing, having usable installed packages/drivers is another.

</div>

Usually the code samples are disregarded when installing new software.

I personally always download them for 2 main reasons:

Check that the installation is successful by running the samples.
Learn about multiple ways the library can be used. It helps in being a playground for the first experiments.

Samples are directly downloaded by following the previous steps.

To compile and use them:

cp -r /usr/local/cuda/samples/ ~/cuda-10_samples

cd ~/cuda-10_samples

make

Finished building CUDA samples

You may get an error like the following:

This can be safely skipped by running:

make -k

This is a know issue in CUDA 10.2 which was not yet solved.

Once built, we try to run one of the examples:

cd ~/cuda-10_samples/bin/x86_64/linux/release
./BlackScholes

...
Test passed

If your tests also passed (hopefully), you can keep going.

If the drivers are installed, the system should be able to see our GPUs, right?

We can call the NVIDIA System Management Interface (nvidia-smi) to check for the available NVIDIA interfaces.

nvidia-smi

There is one weird info here. We installed CUDA 10.2, but nvidia-smi is showing CUDA 11.2, why?

Even when we check the installation location

jean-marc@MSI-GS66:~$ ll /usr/local/
total 44
drwxr-xr-x 11 root root 4096 Jan 29 15:21 ./
drwxr-xr-x 12 root root 4096 Jan 29 17:18 ../
drwxr-xr-x  2 root root 4096 Jan 29 17:24 bin/
lrwxrwxrwx  1 root root    9 Jan 29 15:21 cuda -> cuda-10.2/
drwxr-xr-x 16 root root 4096 Jan 29 14:30 cuda-10.2/
drwxr-xr-x  2 root root 4096 Aug  7 00:34 etc/
drwxr-xr-x  2 root root 4096 Aug  7 00:34 games/
drwxr-xr-x  2 root root 4096 Aug  7 00:34 include/
drwxr-xr-x  5 root root 4096 Jan 29 17:24 lib/
lrwxrwxrwx  1 root root    9 Nov 23 06:04 man -> share/man/
drwxr-xr-x  2 root root 4096 Aug  7 00:34 sbin/
drwxr-xr-x  7 root root 4096 Jan 29 17:19 share/
drwxr-xr-x  2 root root 4096 Aug  7 00:34 src/

We see that only cuda-10.2 is installed, and cuda is a symlink (shortcut) for the cuda-10.2 folder.

It turns out there is a difference between the "GPU CUDA" and the "runtime CUDA".

The CUDA driver API is a low-level API. The CUDA runtime API is a higher-level API that is implemented on top of the CUDA driver API.

You usually use the CUDA runtime API in your code.

The CUDA driver API is installed during the installation of CUDA drivers. The CUDA runtime API is installed during the installation of the CUDA Toolkit.

As long as your CUDA driver API has a version equal to or higher than the CUDA runtime API, you should be good to go with no expected conflicts.

</div>

You can check this great answer on stackoverflow by Robert Crovella, an NVIDIA employee, explaining the difference.

Some details can also be found in the CUDA docs

Once CUDA Toolkit and CUDA Drivers are installed by following the previous steps, we can see our GPUs and the installation seems OK, we can move on to installaing cudNN.

cudNN

What is cudNN

"The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers." - NVIDIA cudNN page

This means that cudNN has highly optimized implementations for common operations used in your applications. We said that the CUDA runtime API can help you leverage GPU parallelization. cudNN can help you use pre-defined functions which are already optimized and using the CUDA runtime API so you don't have to implement them from scratch.

Installation

You need an NVIDIA Developer account for this stage

</div>

As we did for CUDA, we will access the cudNN Archive to select the version we need.

We will download cudNN Library for Linux, and the Code Samples to test our installation. The cudnn-runtime and cudnn-dev packages are also needed.

Once the cudNN tar file and samples are downloaded, navigate to the download directory and follow the subsequent steps:

To install cudNN: Unzip the cuDNN package. x is just a placeholder, choose the correct file downloaded.

tar -xzvf cudnn-x.x-linux-x64-v8.x.x.x.tgz

Copy the following files into the CUDA Toolkit directory, and change the file permissions.

sudo cp cuda/include/cudnn*.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

Install the cudnn-runtime and cudnn-dev deb packages

sudo dpkg -i libcudnn8_8.0.3.33-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn8-dev_8.0.3.33-1+cuda10.2_amd64.deb

To install the samples:

sudo dpkg -i libcudnn8-samples_8.0.3.33-1+cuda10.2_amd64.deb

Verify Installation

We can check the installed version of cudNN

cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR -A 2

#define CUDNN_MAJOR 8
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 3
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#endif /* CUDNN_VERSION_H */

This shows that we have cudNN major version 8, minor version 0, patch version 3, which is the cudNN 8.0.3.

To verify that cuDNN is installed and running properly, compile the mnistCUDNN sample located in the /usr/src/cudnn_samples_v8 directory.

Copy the cuDNN sample to a writable path.

cp -r /usr/src/cudnn_samples_v8/ $HOME

Go to the writable path.

cd  $HOME/cudnn_samples_v8/mnistCUDNN

Compile the mnistCUDNN sample.

make clean && make

Run the mnistCUDNN sample.

./mnistCUDNN

...
Test passed!

If cuDNN is properly installed and running on your Linux system, you will see a message similar to the following: Test passed!

With CUDA and cudNN installed, we still have TensorRT.

TensorRT

What is TensorRT

"The core of NVIDIA TensorRT is a C++ library that facilitates high performance inference on NVIDIA graphics processing units (GPUs). TensorRT takes a trained network, which consists of a network definition and a set of trained parameters, and produces a highly optimized runtime engine which performs inference for that network. TensorRT provides API's via C++ and Python that help to express deep learning models via the Network Definition API or load a pre-defined model via the parsers that allows TensorRT to optimize and run them on an NVIDIA GPU. TensorRT applies graph optimizations, layer fusion, among other optimizations, while also finding the fastest implementation of that model leveraging a diverse collection of highly optimized kernels. TensorRT also supplies a runtime that you can use to execute this network on all of NVIDIA's GPU's from the Kepler generation onwards. TensorRT also includes optional high speed mixed precision capabilities introduced in the Tegra X1, and extended with the Pascal, Volta, and Turing architectures." -NVIDIA Docs

What does that even mean?

Well, this means that TensorRT helps inference of your trained neural networks be much faster. TensorRT will use highly optimized cudNN functions with some additional tricks given your network, to allow a faster inference on NVIDIA GPUs while also using Tensor cores.

Installation

You need an NVIDIA Developer account for this stage

</div>

We can choose the TensorRT version we need by following this link, in our case TensorRT 7.

Then we choose the correct TensorRT version for our CUDA version 10.2.

To install TensorRT:

sudo dpkg -i nv-tensorrt-repo-ubuntu1x04-cudax.x-trt7.x.x.x-ga-yyyymmdd_1-1_amd64.deb

sudo apt-key add /var/nv-tensorrt-repo-cudax.x-trt7.x.x.x-ga-yyyymmdd/7fa2af80.pub

sudo apt-get update

sudo apt-get install tensorrt

We will consider that you may use TensorRT from python2.7 and python3, just in case you face applications still using older versions of Python.

To install additional packages:

For Python2.7

sudo apt-get install python-libnvinfer-dev

For Python3

sudo apt-get install python3-libnvinfer-dev

For additional TensorFlow support

sudo apt-get install uff-converter-tf

Verify Installation

We can check if the packages are correctly installed

dpkg -l | grep nvinfer

Your output should be similar (not necessarly the same) as this

Let's run the TensorRT samples to make sure our installation is successful and usable.

Copy the TensorRT sample to a writable path. (Missing this step would give you a permission error)

cp -r /usr/src/tensorrt/ $HOME/tensorrt-samples

Go to the sampleMNIST example.

cd  $HOME/tensorrt-samples/samples/sampleMNIST

Compile it.

make

Before running our MNIST example, let's download some data for inference. The provided download_pgms.py script uses python3 and the libraries pillow and numpy First we install the libraries and needed tools.

To install python packages, you need Package Installer for Python (pip) to be installed. To install pip for python3

sudo apt install python3-pip python3-dev

To install the packages using pip

python3 -m pip install numpy pillow

We need to install curl, a very powerful tool to transfer data using various network protocols, to download the sample dataset.

sudo apt install curl

Second, we download and extract the data needed from Yann LeCun's website.

cd $HOME/tensorrt-samples/data/mnist

curl http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz -o train-images-idx3-ubyte.gz

gunzip train-images-idx3-ubyte.gz

curl http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz -o train-labels-idx1-ubyte.gz

gunzip train-labels-idx1-ubyte.gz

Third, we run the data download script

python3 download_pgms.py

Finally, we run the MNIST sample.

cd $HOME/tensorrt-samples/bin && ./sample_mnist

&&&& RUNNING TensorRT.sample_mnist # ./sample_mnist
[08/21/2020-10:51:14] [I] Building and running a GPU inference engine for MNIST
[08/21/2020-10:51:15] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[08/21/2020-10:51:15] [I] Input:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@#=.  +*=#@@@@@@@
@@@@@@@@@@@*   :.   -@@@@@@@
@@@@@@@@@@#  :#@@:  +@@@@@@@
@@@@@@@@@*  :@@@*  .@@@@@@@@
@@@@@@@@=  =@@@@.  *@@@@@@@@
@@@@@@@=  -@@@@*  =@@@@@@@@@
@@@@@@@  -@@@%:  -@@@@@@@@@@
@@@@@@%  %%+:    *@@@@@@@@@@
@@@@@@@      ..  @@@@@@@@@@@
@@@@@@@#  .=%%: =@@@@@@@@@@@
@@@@@@@@@@@@@#  +@@@@@@@@@@@
@@@@@@@@@@@@@#  @@@@@@@@@@@@
@@@@@@@@@@@@@@  @@@@@@@@@@@@
@@@@@@@@@@@@@#  @@@@@@@@@@@@
@@@@@@@@@@@@@+  @@@@@@@@@@@@
@@@@@@@@@@@@@%  @@@@@@@@@@@@
@@@@@@@@@@@@@@. #@@@@@@@@@@@
@@@@@@@@@@@@@@* :%@@@@@@@@@@
@@@@@@@@@@@@@@@: -@@@@@@@@@@
@@@@@@@@@@@@@@@@= %@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[08/21/2020-10:51:15] [I] Output:
0:
1:
2:
3:
4:
5:
6:
7:
8:
9: **********

&&&& PASSED TensorRT.sample_mnist # ./sample_mnist

It is alive! Given a picture of a 9, we were able to predict that it is the actual digit 9 (** next to the 9). You may have different numbers, the important part is seeing "PASSED" with a similar output to this one.

Conclusion

Congratulations! Your system is now ready to be used for many applications which leverage your NVIDIA GPU. If you are interested in CUDA, cudNN, and TensorRT, you have the samples ready to use!

References

Make sure to check the version of the documentation. Sometimes it points to the latest version only and it may not be compatible with the version you need!

</div>