Jekyll2018-01-04T16:51:32+00:00http://notes.jnadeau.ca/Julian’s NotesNotes about various topics. Mostly computer science related.Julian NadeauDeveloper Productivity2018-01-03T17:13:05+00:002018-01-03T17:13:05+00:00http://notes.jnadeau.ca/computers/2018/01/03/dev_productivity<h2 id="what-is-developer-productivity">What is Developer Productivity?</h2>
<p>Productivity is defined as “the effectiveness of productive effort, especially in industry, as measured in terms of the rate of output per unit of input.” Therefore, developer Productivity can be described as a concept, set of tools or processes, or a team that is dedicated to enhancing the efficiency of other developers with the goal of allowing them to increase their overall output.</p>
<p>This concept and the responsibilities of the team in charge of developer productivity can be varied throughout different companies, however after a number of meetings it became clear that the teams mostly focus on a few key areas. Unfortunately, the developers at different companies don’t seem to discuss, have meet-ups, or work together on allied goals.</p>
<p>Developer productivity is not a business concern nor is most of the work confidential. The work is generally generic and the concepts are easily shared and discussed. Over the course of many meetings with similar teams at other companies, it became obvious that we’re all duplicating each others’ work and more importantly we’re duplicating the exploration.</p>
<h2 id="vision">Vision</h2>
<p>Developer Productivity is not a business concern and it is not specific to any company. The concepts are shareable and can help reduce time to explore, gain more feedback and expertise, and allow us to help define standards in the industry.</p>
<p>If we work together in creating a community and a set of community guidelines - we will all benefit.</p>
<h2 id="phase-i">Phase I</h2>
<p>Phase I is simply the initial foundation of the community. The Slack channel, invitations at <a href="https://chat.devproductivity.io">https://chat.devproductivity.io</a>, is a good first step towards an initial foundation.</p>
<p>As we grow the audience, we can start to hold online meet-ups. The first one is currently scheduled for sometime in January 2018.</p>
<p>These simple ideas will allow us to start to grow a community and provide the foundations to introduce work on additional phases of the project.</p>
<blockquote>
<p><strong>Success Criteria:</strong> The community starts to grow and people remain excited. There is a decent turnout for the online meet-ups and people are asking what comes next.</p>
</blockquote>
<h2 id="phase-ii">Phase II</h2>
<p>There is no current ground work laid to bootstrap a community dedicated to developer productivity. This means that we have the opportunity to create this community and share the work we do with each other.</p>
<p><a href="https://devproductivity.io">https://devproductivity.io</a> will be a central hub managed by the community. With links to content about continuous integration and testing, automation, operational excellence, developer environments, mobile tooling, and other aspects of developer productivity - this website will be a central location on which to grow a solid foundation.</p>
<p>This website/community allows us to draw talent and work from the larger pool of engineers working on the same goals, it also allows us to share our ideas, and educate the larger developer community about standard goals.</p>
<p>This website is envisioned to be a handbook or styleguide for all people working on developer productivity. With a website similar to <a href="https://fastlane.tools">https://fastlane.tools</a> or <a href="https://polaris.shopify.com">https://polaris.shopify.com</a>, we will gain the ability to communicate industry best practices and crowd source their definitions. Moreover, we can expand the website to include blogs (written by the community members!), useful papers and links, forums, and other useful materials.</p>
<blockquote>
<p><strong>Success Criteria:</strong> The community gets excited about the website and we start to see traffic pick up. The community will start to become more involved and take ownership of parts of the content once the initial site is launched.</p>
</blockquote>
<h2 id="phase-iii">Phase III</h2>
<p>As we grow, online meet-ups will likely not scale and people will want to have more meaningful in person collaborations and discussions. The website will need more community to continue to scale and continue to be a place that people frequent.</p>
<p>In Phase III, we hold our first conference. Development has had conferences of so many varieties, but there has never been a conference dedicated to the concepts that form developer productivity. That being said some conferences, such as Velocity and DevDaysTO, dedicate portions of their content towards developer productivity but no known conference is targetted specifically towards developer productivity.</p>
<p>By holding the conference, we aim to become a driving force behind Developer Productivity and increase community knowledge and commitment to this community.</p>
<blockquote>
<p><strong>Success Criteria:</strong> We see excitement and participation. The website and the content continues to increase in traffic and a vibrant community is formed.</p>
</blockquote>
<h2 id="phase-n">Phase N</h2>
<p>We may not need to go much further than a conference. If these phases are successful, we may have found a recipe to continue to grow and continue to be a power house in the developer productivity community.</p>
<p>We also don’t have all of the ideas and don’t want to be prescriptive about how the community grows. We want to see how it naturally grows and in what direction the community wants to grow.</p>Julian NadeauWhat is Developer Productivity? Productivity is defined as “the effectiveness of productive effort, especially in industry, as measured in terms of the rate of output per unit of input.” Therefore, developer Productivity can be described as a concept, set of tools or processes, or a team that is dedicated to enhancing the efficiency of other developers with the goal of allowing them to increase their overall output. This concept and the responsibilities of the team in charge of developer productivity can be varied throughout different companies, however after a number of meetings it became clear that the teams mostly focus on a few key areas. Unfortunately, the developers at different companies don’t seem to discuss, have meet-ups, or work together on allied goals. Developer productivity is not a business concern nor is most of the work confidential. The work is generally generic and the concepts are easily shared and discussed. Over the course of many meetings with similar teams at other companies, it became obvious that we’re all duplicating each others’ work and more importantly we’re duplicating the exploration. Vision Developer Productivity is not a business concern and it is not specific to any company. The concepts are shareable and can help reduce time to explore, gain more feedback and expertise, and allow us to help define standards in the industry. If we work together in creating a community and a set of community guidelines - we will all benefit. Phase I Phase I is simply the initial foundation of the community. The Slack channel, invitations at https://chat.devproductivity.io, is a good first step towards an initial foundation. As we grow the audience, we can start to hold online meet-ups. The first one is currently scheduled for sometime in January 2018. These simple ideas will allow us to start to grow a community and provide the foundations to introduce work on additional phases of the project. Success Criteria: The community starts to grow and people remain excited. There is a decent turnout for the online meet-ups and people are asking what comes next. Phase II There is no current ground work laid to bootstrap a community dedicated to developer productivity. This means that we have the opportunity to create this community and share the work we do with each other. https://devproductivity.io will be a central hub managed by the community. With links to content about continuous integration and testing, automation, operational excellence, developer environments, mobile tooling, and other aspects of developer productivity - this website will be a central location on which to grow a solid foundation. This website/community allows us to draw talent and work from the larger pool of engineers working on the same goals, it also allows us to share our ideas, and educate the larger developer community about standard goals. This website is envisioned to be a handbook or styleguide for all people working on developer productivity. With a website similar to https://fastlane.tools or https://polaris.shopify.com, we will gain the ability to communicate industry best practices and crowd source their definitions. Moreover, we can expand the website to include blogs (written by the community members!), useful papers and links, forums, and other useful materials. Success Criteria: The community gets excited about the website and we start to see traffic pick up. The community will start to become more involved and take ownership of parts of the content once the initial site is launched. Phase III As we grow, online meet-ups will likely not scale and people will want to have more meaningful in person collaborations and discussions. The website will need more community to continue to scale and continue to be a place that people frequent. In Phase III, we hold our first conference. Development has had conferences of so many varieties, but there has never been a conference dedicated to the concepts that form developer productivity. That being said some conferences, such as Velocity and DevDaysTO, dedicate portions of their content towards developer productivity but no known conference is targetted specifically towards developer productivity. By holding the conference, we aim to become a driving force behind Developer Productivity and increase community knowledge and commitment to this community. Success Criteria: We see excitement and participation. The website and the content continues to increase in traffic and a vibrant community is formed. Phase N We may not need to go much further than a conference. If these phases are successful, we may have found a recipe to continue to grow and continue to be a power house in the developer productivity community. We also don’t have all of the ideas and don’t want to be prescriptive about how the community grows. We want to see how it naturally grows and in what direction the community wants to grow.Wakame (Seaweed)2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/culinary/2017/04/05/wakame<p><img src="http://img.mindbodygreen.com/image/upload/c_limit,w_350,f_auto/ftr/wakame.jpg" /></p>
<p>A staple in Japanese cuisine, Wakame (ワカメ or Undaria pinnatifida) is a sea vegetable/edible seaweed.</p>
<p>A subtly sweet flavour that is ripe with umami. It is usually very salty too. It has a satiny texture.</p>
<p>The leaves expand during cooking, so cut the pieces up with that in mind.</p>
<h2 id="uses">Uses</h2>
<ul>
<li>Wakame Salad (seaweed salad)</li>
<li>Toppings for sandwiches, meat dishes, etc</li>
<li>Soups</li>
<li>Side dish</li>
</ul>
<h2 id="minerals-and-nutrients">Minerals and Nutrients</h2>
<h3 id="watch-out-for">Watch out for</h3>
<ul>
<li>High in sodium</li>
</ul>
<h3 id="good-for">Good for</h3>
<p>Wakame is low in calories and is a great source of vitamins and minerals. It includes:</p>
<ul>
<li>iodine</li>
<li>iron</li>
<li>calcium</li>
<li>magnesium</li>
<li>folate</li>
<li>vitamin A</li>
<li>vitamin C</li>
<li>vitamin D</li>
<li>vitamin E</li>
<li>vitamin K</li>
<li>vitamin B2</li>
<li>lignans</li>
<li>fucoxanthin</li>
<li>eicosapentaenoic acid, an omega-3 fatty acid</li>
</ul>
<h2 id="nutritional-information">Nutritional Information</h2>
<p>Values are per 100g</p>
<h3 id="overview">Overview</h3>
<table>
<thead>
<tr>
<th> </th>
<th>Amount</th>
</tr>
</thead>
<tbody>
<tr>
<td>Calories</td>
<td>45</td>
</tr>
<tr>
<td>Carbohydrates</td>
<td>9.14g</td>
</tr>
<tr>
<td>Sugars</td>
<td>0.65g</td>
</tr>
<tr>
<td>Dietary Fiber</td>
<td>0.5g</td>
</tr>
<tr>
<td>Fat</td>
<td>0.64g</td>
</tr>
<tr>
<td>Protein</td>
<td>3.03g</td>
</tr>
</tbody>
</table>
<h3 id="vitamins">Vitamins</h3>
<table>
<thead>
<tr>
<th>Vitamin</th>
<th>Percent</th>
<th>Amount</th>
</tr>
</thead>
<tbody>
<tr>
<td>Thiamine (B1)</td>
<td>5%</td>
<td>0.06 mg</td>
</tr>
<tr>
<td>Riboflavin (B2)</td>
<td>19%</td>
<td>0.23 mg</td>
</tr>
<tr>
<td>Niacin (B3)</td>
<td>11%</td>
<td>1.6 mg</td>
</tr>
<tr>
<td>Pantothenic acid (B5)</td>
<td>14%</td>
<td>0.697 mg</td>
</tr>
<tr>
<td>Folate (B9)</td>
<td>49%</td>
<td>196 μg</td>
</tr>
<tr>
<td>Vitamin C</td>
<td>4%</td>
<td>3 mg</td>
</tr>
<tr>
<td>Vitamin E</td>
<td>7%</td>
<td>1 mg</td>
</tr>
<tr>
<td>Vitamin K</td>
<td>5%</td>
<td>5.3 μg</td>
</tr>
</tbody>
</table>
<h3 id="minerals">Minerals</h3>
<table>
<thead>
<tr>
<th>Mineral</th>
<th>Percent</th>
<th>Amount</th>
</tr>
</thead>
<tbody>
<tr>
<td>Calcium</td>
<td>15%</td>
<td>150 mg</td>
</tr>
<tr>
<td>Iron</td>
<td>17%</td>
<td>2.18 mg</td>
</tr>
<tr>
<td>Magnesium</td>
<td>30%</td>
<td>107 mg</td>
</tr>
<tr>
<td>Manganese</td>
<td>67%</td>
<td>1.4 mg</td>
</tr>
<tr>
<td>Phosphorus</td>
<td>11%</td>
<td>80 mg</td>
</tr>
<tr>
<td>Sodium</td>
<td>58%</td>
<td>872 mg</td>
</tr>
<tr>
<td>Zinc</td>
<td>4%</td>
<td>0.38 mg</td>
</tr>
</tbody>
</table>Julian NadeauA staple in Japanese cuisine, Wakame (ワカメ or Undaria pinnatifida) is a sea vegetable/edible seaweed. A subtly sweet flavour that is ripe with umami. It is usually very salty too. It has a satiny texture. The leaves expand during cooking, so cut the pieces up with that in mind. Uses Wakame Salad (seaweed salad) Toppings for sandwiches, meat dishes, etc Soups Side dish Minerals and Nutrients Watch out for High in sodium Good for Wakame is low in calories and is a great source of vitamins and minerals. It includes: iodine iron calcium magnesium folate vitamin A vitamin C vitamin D vitamin E vitamin K vitamin B2 lignans fucoxanthin eicosapentaenoic acid, an omega-3 fatty acid Nutritional Information Values are per 100g Overview Amount Calories 45 Carbohydrates 9.14g Sugars 0.65g Dietary Fiber 0.5g Fat 0.64g Protein 3.03g Vitamins Vitamin Percent Amount Thiamine (B1) 5% 0.06 mg Riboflavin (B2) 19% 0.23 mg Niacin (B3) 11% 1.6 mg Pantothenic acid (B5) 14% 0.697 mg Folate (B9) 49% 196 μg Vitamin C 4% 3 mg Vitamin E 7% 1 mg Vitamin K 5% 5.3 μg Minerals Mineral Percent Amount Calcium 15% 150 mg Iron 17% 2.18 mg Magnesium 30% 107 mg Manganese 67% 1.4 mg Phosphorus 11% 80 mg Sodium 58% 872 mg Zinc 4% 0.38 mgumami2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/culinary/2017/04/05/umami<h2 id="umami">Umami</h2>
<p>(/uˈmɑːmi/)</p>
<p>Umami is also known as the “savoury taste” and is one of the five basic tastes. It is described as brothy or meaty.</p>
<p>People taste umami using taste receptors for glutamate (hence why monosodium glutamate [MSG] is essentially pure umami).</p>
<h3 id="wip">WIP</h3>Julian NadeauUmami (/uˈmɑːmi/) Umami is also known as the “savoury taste” and is one of the five basic tastes. It is described as brothy or meaty. People taste umami using taste receptors for glutamate (hence why monosodium glutamate [MSG] is essentially pure umami). WIPbundler/setup2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/computers/2017/04/05/setup<p>Bundler setup parses through dependencies and compiles them into proper load paths. This step, on smaller applications, takes very little time. However on larger applications, this step can take a long duration - about 700-750ms to be exact.</p>
<p>Below are notes about how long certain parts take.</p>
<h3 id="timing-helper">Timing Helper</h3>
<p>Throughtout these notes, I am using a method <code class="highlighter-rouge">_t</code>. This is a timing helper for scrappy timing defined as such:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">_t</span><span class="p">(</span><span class="n">label</span><span class="p">)</span>
<span class="n">t</span> <span class="o">=</span> <span class="no">Process</span><span class="p">.</span><span class="nf">clock_gettime</span><span class="p">(</span><span class="no">Process</span><span class="o">::</span><span class="no">CLOCK_MONOTONIC</span><span class="p">)</span>
<span class="n">ret</span> <span class="o">=</span> <span class="k">yield</span>
<span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="n">label</span><span class="si">}</span><span class="s2"> </span><span class="si">#{</span><span class="no">Process</span><span class="p">.</span><span class="nf">clock_gettime</span><span class="p">(</span><span class="no">Process</span><span class="o">::</span><span class="no">CLOCK_MONOTONIC</span><span class="p">)</span> <span class="o">-</span> <span class="n">t</span><span class="si">}</span><span class="s2">"</span>
<span class="n">ret</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The key thing to note is that it uses CPU time and the return value is whatever it is from the yield. The latter point makes it easy to track things down.</p>
<h2 id="highest-level">Highest Level</h2>
<p>If we open the <code class="highlighter-rouge">bundler/setup.rb</code> file up, we might notice that it is small enough to simply benchmark each line. Doing this results in the following sequence diagram:</p>
<!---
```diagram
gantt
title require 'bundler/setup'
dateFormat s.SSS
section require
bundler/postit_trampoline :a1, 0.000, 0.006
bundler/shared_helpers :a1, 0.006, 0.007
bundler :a1, 0.007, 0.010
section Bundler
Bundler.setup :a2, 0.010, 0.710
section various
"other" :a3, 0.710, 0.711
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/ab3cdf34c521c668b44359644dcd6d8f.png" alt="diagram image" height="400px" /></p>
<p>We can take note that <code class="highlighter-rouge">Bundler.setup</code> results in almost the entire duration of the call to <code class="highlighter-rouge">require 'bundler/setup'</code>. Let’s dig into that more.</p>
<h2 id="bundlersetup">Bundler.setup</h2>
<p>The call to <code class="highlighter-rouge">Bundler.setup</code> is a little bit ambiguous due to parameters, but checking the <code class="highlighter-rouge">source_location</code> at runtime results in <code class="highlighter-rouge">setup</code> at line 90 of <code class="highlighter-rouge">lib/bundler.rb</code>.
This was what I originally thought, but it it good to check.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Bundler</span><span class="p">.</span><span class="nf">method</span><span class="p">(</span><span class="ss">:setup</span><span class="p">).</span><span class="nf">source_location</span>
<span class="p">[</span><span class="s2">"/Users/juliannadeau/.gem/ruby/2.3.3/gems/bundler-1.14.5/lib/bundler.rb"</span><span class="p">,</span> <span class="mi">90</span><span class="p">]</span>
</code></pre></div></div>
<p>The method definition here is as follows:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">return</span> <span class="vi">@setup</span> <span class="k">if</span> <span class="n">defined?</span><span class="p">(</span><span class="vi">@setup</span><span class="p">)</span> <span class="o">&&</span> <span class="vi">@setup</span>
<span class="n">definition</span><span class="p">.</span><span class="nf">validate_runtime!</span>
<span class="no">SharedHelpers</span><span class="p">.</span><span class="nf">print_major_deprecations!</span>
<span class="k">if</span> <span class="n">groups</span><span class="p">.</span><span class="nf">empty?</span>
<span class="c1"># Load all groups, but only once</span>
<span class="vi">@setup</span> <span class="o">=</span> <span class="nb">load</span><span class="p">.</span><span class="nf">setup</span>
<span class="k">else</span>
<span class="nb">load</span><span class="p">.</span><span class="nf">setup</span><span class="p">(</span><span class="o">*</span><span class="n">groups</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We can see that it caches the orginal result on the <code class="highlighter-rouge">Bundler</code> class and so we can only call it once per run. This is good as it will save a lot of time if we happen to call it twice.</p>
<p>A few questions I have up front:</p>
<ul>
<li>is <code class="highlighter-rouge">definition</code> a variable or a method? Given that this is the first call to a class, it’s probably a method.</li>
<li><code class="highlighter-rouge">groups</code> is almost definitely empty. It is probably a method too. Is it cached?</li>
<li>same thing with <code class="highlighter-rouge">load</code></li>
</ul>
<p>The reason this is important is that while the method calls on the return values of the methods mentioned above should be traced, we need to make sure that the method calls to get those return values
aren’t slow either. To do this, we will need to split up the variable/method calls.</p>
<p>We end up with this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">return</span> <span class="vi">@setup</span> <span class="k">if</span> <span class="n">defined?</span><span class="p">(</span><span class="vi">@setup</span><span class="p">)</span> <span class="o">&&</span> <span class="vi">@setup</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">_t</span><span class="p">(</span><span class="s1">'definition'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">definition</span>
<span class="k">end</span>
<span class="n">_t</span><span class="p">(</span><span class="s1">'validate_runtime!'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">d</span><span class="p">.</span><span class="nf">validate_runtime!</span>
<span class="k">end</span>
<span class="n">_t</span><span class="p">(</span><span class="s1">'print_major_deprecations'</span><span class="p">)</span> <span class="k">do</span>
<span class="no">SharedHelpers</span><span class="p">.</span><span class="nf">print_major_deprecations!</span>
<span class="k">end</span>
<span class="n">g</span> <span class="o">=</span> <span class="n">_t</span><span class="p">(</span><span class="s1">'groups'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">groups</span>
<span class="k">end</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">_t</span><span class="p">(</span><span class="s1">'load'</span><span class="p">)</span> <span class="k">do</span>
<span class="nb">load</span>
<span class="k">end</span>
<span class="k">if</span> <span class="n">g</span><span class="p">.</span><span class="nf">empty?</span>
<span class="c1"># Load all groups, but only once</span>
<span class="vi">@setup</span> <span class="o">=</span> <span class="n">_t</span><span class="p">(</span><span class="s1">'setup 1'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">l</span><span class="p">.</span><span class="nf">setup</span>
<span class="k">end</span>
<span class="k">else</span>
<span class="n">_t</span><span class="p">(</span><span class="s1">'setup 2'</span><span class="p">)</span> <span class="p">{</span> <span class="n">l</span><span class="p">.</span><span class="nf">setup</span><span class="p">(</span><span class="o">*</span><span class="n">groups</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="results-of-timing">Results of timing</h3>
<!---
```diagram
gantt
title Bundler.setup
dateFormat s.SSS
section definition
initialize :a1, 0.000, 0.129
definition.validate_runtime! :a1, 0.129, 0.130
section SharedHelpers
print_major_deprecations! :a2, 0.130, 0.131
section groups
groups :a3, 0.131, 0.132
section load
load :a4, 0.132, 0.133
load.setup :a4, 0.133, 0.683
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/75890057a20de01f006baac5a4c816ab.png" alt="diagram image" height="400px" /></p>
<p>It is painfully obvious that we spend a lot of time in 2 spots. About 1/3 of the time is spent in <code class="highlighter-rouge">definition</code>, and the other 2/3 is spent in <code class="highlighter-rouge">load.setup</code> (specifically the <code class="highlighter-rouge">setup</code> call). We’ll dig into both of these separately.</p>
<hr />
<p>To continue this path:</p>
<ul>
<li><a href="../definition">definition</a></li>
<li><a href="../load">load</a></li>
</ul>Julian NadeauBundler setup parses through dependencies and compiles them into proper load paths. This step, on smaller applications, takes very little time. However on larger applications, this step can take a long duration - about 700-750ms to be exact. Below are notes about how long certain parts take. Timing Helper Throughtout these notes, I am using a method _t. This is a timing helper for scrappy timing defined as such: def _t(label) t = Process.clock_gettime(Process::CLOCK_MONOTONIC) ret = yield puts "#{label} #{Process.clock_gettime(Process::CLOCK_MONOTONIC) - t}" ret end The key thing to note is that it uses CPU time and the return value is whatever it is from the yield. The latter point makes it easy to track things down. Highest Level If we open the bundler/setup.rb file up, we might notice that it is small enough to simply benchmark each line. Doing this results in the following sequence diagram: We can take note that Bundler.setup results in almost the entire duration of the call to require 'bundler/setup'. Let’s dig into that more. Bundler.setup The call to Bundler.setup is a little bit ambiguous due to parameters, but checking the source_location at runtime results in setup at line 90 of lib/bundler.rb. This was what I originally thought, but it it good to check. Bundler.method(:setup).source_location ["/Users/juliannadeau/.gem/ruby/2.3.3/gems/bundler-1.14.5/lib/bundler.rb", 90] The method definition here is as follows: return @setup if defined?(@setup) && @setup definition.validate_runtime! SharedHelpers.print_major_deprecations! if groups.empty? # Load all groups, but only once @setup = load.setup else load.setup(*groups) end We can see that it caches the orginal result on the Bundler class and so we can only call it once per run. This is good as it will save a lot of time if we happen to call it twice. A few questions I have up front: is definition a variable or a method? Given that this is the first call to a class, it’s probably a method. groups is almost definitely empty. It is probably a method too. Is it cached? same thing with load The reason this is important is that while the method calls on the return values of the methods mentioned above should be traced, we need to make sure that the method calls to get those return values aren’t slow either. To do this, we will need to split up the variable/method calls. We end up with this: return @setup if defined?(@setup) && @setup d = _t('definition') do definition end _t('validate_runtime!') do d.validate_runtime! end _t('print_major_deprecations') do SharedHelpers.print_major_deprecations! end g = _t('groups') do groups end l = _t('load') do load end if g.empty? # Load all groups, but only once @setup = _t('setup 1') do l.setup end else _t('setup 2') { l.setup(*groups) } end Results of timing It is painfully obvious that we spend a lot of time in 2 spots. About 1/3 of the time is spent in definition, and the other 2/3 is spent in load.setup (specifically the setup call). We’ll dig into both of these separately. To continue this path: definition loadRubyGems2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/computers/2017/04/05/ruby_gems<h2 id="specification">Specification</h2>
<!---
```diagram
gantt
title file: /opt/rubies/2.3.3/lib/ruby/site_ruby/2.3.0/rubygems/specification.rb method: load
numberFormat %.2f
"_spec = LOAD_CACHE[file] (run 296 times)" :a1, 0.000, 0.728
"return _spec if _spec (run 295 times)" :a1, 0.728, 1.456
"file = file.dup.untaint (run 295 times)" :a1, 1.456, 2.183
"return unless File.file?(file) (run 295 times)" :a1, 2.183, 2.911
"code = if defined? Encoding (run 295 times)" :a1, 2.911, 3.639
"File.read file :mode => 'r:UTF-8:-' (run 295 times)" :a1, 3.639, 4.367
"code.untaint (run 295 times)" :a1, 4.367, 5.095
"begin (run 295 times)" :a1, 5.095, 5.823
"_spec = eval code binding file (run 295 times)" :a1, 5.823, 97.089
"if Gem::Specification === _spec (run 295 times)" :a1, 97.089, 97.817
"_spec.loaded_from = File.expand_path file.to_s (run 295 times)" :a1, 97.817, 98.544
"LOAD_CACHE[file] = _spec (run 295 times)" :a1, 98.544, 99.272
"return _spec" :a1, 99.272, 100.000
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/f50cd68abdc716c81b609381352d8c7e.png" alt="diagram image" width="100%" /></p>Julian NadeauSpecificationRails Autoloading2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/computers/2017/04/05/rails_autoloading<p>Autoloading code is a mechanism in Rails that causes frameworks, classes, and code to be loaded automatically on boot. This helps productivity by allowing developers to freely use constants and classes without having to explicitly require them.</p>
<p>An issue arises however that large amounts of code that are not needed for boot are loaded during the boot of an application, or are loaded out of order.</p>
<p>The diagram below shows how files and classes are autoloaded.</p>
<!---
```diagram
graph TD
subgraph Autoloading
Autoload
Finished
end
subgraph AutoloadPath
AutoloadPath
NameError
end
subgraph Loading
Load
LoadError
end
subgraph Parsing
Parse
end
%% Autoloading
Entry[Start Here]-\->Autoload
Autoload-- Empty Autoload Path -\->Finished
Autoload-- Load path from autoload path -\->Load
%% AutoloadPath Paths
AutoloadPath--Cannot find a class to match Constant -\->NameError[NameError: uninitialized constant MyConstant]
AutoloadPath-- Find file that matches the Constant -\->Load[Load File]
%% Parse Paths
Parse-- Encounter Constant we don't know -\->AutoloadPath
Parse-. Finished Parsing .->Autoload
%% Load Paths
Load-- Class definition matches file -\->Parse[Parse Class]
Load-- Class definition does not match file -\->LoadError[LoadError: Expected `file` to define Class]
%% Load-. Finished Loading Class .->Autoload
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/0f5e2e3da6b82b2ac0c974d9c82e0297.png" alt="diagram image" class="full-width" /></p>
<h3 id="problem">Problem</h3>
<p>Load order dependency issues can happen due to nested class defintions.</p>
<p>In the code snippet below, class <code class="highlighter-rouge">A</code> defines a class <code class="highlighter-rouge">B</code>. This means that the constant <code class="highlighter-rouge">B</code> is now defined. In the diagram above, we see that the un-nested class <code class="highlighter-rouge">B</code> depends on the <code class="highlighter-rouge">ConstantMissing</code> error to load it during auto-load. However, since <code class="highlighter-rouge">A::B</code> is defined, a <code class="highlighter-rouge">ConstantMissing</code> hook will never happen as <code class="highlighter-rouge">B</code> will resolve to <code class="highlighter-rouge">A::B</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">A</span>
<span class="k">class</span> <span class="nc">B</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">B</span>
<span class="k">end</span>
</code></pre></div></div>
<p>In particular, from the diagram above, this part never happens.</p>
<p><img src="https://jules2689.github.io/gitcdn/images/website/doesnt_happen.png" alt="diagram image" /></p>Julian NadeauAutoloading code is a mechanism in Rails that causes frameworks, classes, and code to be loaded automatically on boot. This helps productivity by allowing developers to freely use constants and classes without having to explicitly require them. An issue arises however that large amounts of code that are not needed for boot are loaded during the boot of an application, or are loaded out of order. The diagram below shows how files and classes are autoloaded. Problem Load order dependency issues can happen due to nested class defintions. In the code snippet below, class A defines a class B. This means that the constant B is now defined. In the diagram above, we see that the un-nested class B depends on the ConstantMissing error to load it during auto-load. However, since A::B is defined, a ConstantMissing hook will never happen as B will resolve to A::B. class A class B end end class B end In particular, from the diagram above, this part never happens.Caching Paths2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/computers/2017/04/05/path_scanner<hr />
<ul>
<li><a href="../bootsnap">Overview</a></li>
<li><a href="../caching_paths">Caching Paths</a></li>
<li><a href="../path_scanner">Path Scanner</a></li>
</ul>
<hr />
<p>The Path Scanner is intended to identify all files and folders within a given path that are not in the bundler path already. As a result, we can then use this result to cache path loading.</p>
<!---
```diagram
graph TD
StartPoint[Starting Point]-\->Relative?
Relative?[is path relative?]--yes-\->Error[raise RelativePathNotSupported error]
Relative?--no-\->DirListing[iterator for all requirables from path]
DirListing--Next entry-\->DescendentOfBundlePath[bundle path is a descendent of this path?**]
DirListing--No next entry-\->Return[return dirs and requireables]
subgraph Directory Glob
DescendentOfBundlePath--yes-\->DirListing
DescendentOfBundlePath--no-\->Dir?
AddDir-\->DirListing
AddRequireable-\->DirListing
Dir?--yes-\->AddDir[Add to dirs]
Dir?--no-\->AddRequireable[Add to requireables]
end
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/9cb9b39adf4bf924c16adbb0ef246aa4.png" alt="diagram image" width="100%" /></p>
<p>** If the bundle path is a descendent of this path, we do additional checks to prevent recursing into the bundle path as we recurse through this path. We don’t want to scan the bundle path because anything useful in</p>Julian NadeauOverview Caching Paths Path Scanner The Path Scanner is intended to identify all files and folders within a given path that are not in the bundler path already. As a result, we can then use this result to cache path loading. ** If the bundle path is a descendent of this path, we do additional checks to prevent recursing into the bundle path as we recurse through this path. We don’t want to scan the bundle path because anything useful inMoving Average Convergence Divergence - MACD2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/finance/2017/04/05/macd<p>It is a trend following momentum indicator showing the relationship between 2 moving averages.</p>
<p>The MACD is calculated by:</p>
<ul>
<li>subtracting the 26-day exponential moving average (EMA) from the 12-day EMA.</li>
<li>A 9-day EMA of the MACD is plotted on top of this.</li>
<li>It is used as a signal line to indicate when to buy and sell.</li>
</ul>
<p><img src="http://i.investopedia.com/inv/dictionary/terms/macd1.gif" width="300" /></p>
<h2 id="interpretation">Interpretation</h2>
<h3 id="crossovers">Crossovers</h3>
<p>When the MACD falls below the signal line, it is a “bearish” signal which indicates that it may be time to sell.</p>
<p>Conversely, when it rises above the signal line, it may indicate an upward momentum.</p>
<h3 id="divergence">Divergence</h3>
<p>When the price diverges from the MACD, it means the end of the current trend.</p>
<h3 id="a-dramatic-rise">A Dramatic Rise</h3>
<p>When the shorter term (9 day EMA) pulls away from the longer term (26 day EMA) it means that the stock is overbought and will soon return to normal levels.</p>
<h3 id="other">Other</h3>
<p>When the line moves above or below the zero line, this is a signal the position of the short term average relative to the long term average.</p>
<p>When it is above zero, the short term average is above the long term average. This indicates upward momentum. When it is below zero, it indicates downward momentum.</p>Julian NadeauIt is a trend following momentum indicator showing the relationship between 2 moving averages. The MACD is calculated by: subtracting the 26-day exponential moving average (EMA) from the 12-day EMA. A 9-day EMA of the MACD is plotted on top of this. It is used as a signal line to indicate when to buy and sell. Interpretation Crossovers When the MACD falls below the signal line, it is a “bearish” signal which indicates that it may be time to sell. Conversely, when it rises above the signal line, it may indicate an upward momentum. Divergence When the price diverges from the MACD, it means the end of the current trend. A Dramatic Rise When the shorter term (9 day EMA) pulls away from the longer term (26 day EMA) it means that the stock is overbought and will soon return to normal levels. Other When the line moves above or below the zero line, this is a signal the position of the short term average relative to the long term average. When it is above zero, the short term average is above the long term average. This indicates upward momentum. When it is below zero, it indicates downward momentum.Logical Clocks2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/computers/2017/04/05/logical_clocks<p><a href="http://web.cs.iastate.edu/~cs554/NOTES/Ch6-LogicalClocks.pdf">This</a> is a great presentation.</p>
<p>Logical clocks are used to agree on order in which events occur. The absolute/real time is not important in this concept.</p>
<p>Event ordering can be based on any number of factors. In a local system, CPU time can be used. But in a distributed system, there is no perfectly synchronized time or clock that can be used, and local times may not be in sync (and probably are not). <a href="https://en.wikipedia.org/wiki/Leslie_Lamport">Lamport</a> suggested a logical clock be used to address this.</p>
<h4 id="key-concepts">Key concepts</h4>
<ul>
<li>Processes exchange messages</li>
<li>Message must be sent before received</li>
<li>Send/receive used to order events and synchronize logical clocks</li>
</ul>
<h4 id="properties">Properties</h4>
<ul>
<li>If A happens before B in the same process (or system), then <code class="highlighter-rouge">A -> B</code></li>
<li><code class="highlighter-rouge">A -> B</code> also means that A sent the message and B means the receipt of it</li>
<li>Relation is transitive: e.g <code class="highlighter-rouge">A -> B</code> and <code class="highlighter-rouge">B -> C</code> implies <code class="highlighter-rouge">A -> C</code></li>
<li>Unordered events are concurrent: <code class="highlighter-rouge">A !-> B</code> and <code class="highlighter-rouge">B !-></code> A implies <code class="highlighter-rouge">A || B</code></li>
</ul>
<h2 id="lamports-logical-clocks">Lamport’s Logical Clocks</h2>
<ul>
<li>If <code class="highlighter-rouge">A -> B</code> then <code class="highlighter-rouge">timestamp(A) < timestamp(B)</code></li>
</ul>
<h3 id="lamports-algorithm">Lamport’s Algorithm</h3>
<!---
```diagram
sequenceDiagram
Note left of i: Logical Clock: L(i)
Note right of j: Logical Clock: L(j)
loop Every Event: i => j
i->>i: Event Occurs. L(i) = L(i) + 1
i->>j: Event Sent. Send L(i)
j->>j: Event Received. L(j) = MAX(L(i), L(J)) + 1
end
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/8df698ce798b092930d7fb6955a38bc6.png" alt="diagram image" height="450px" /></p>
<p><strong>Note</strong>: <code class="highlighter-rouge">A -> B</code> implies <code class="highlighter-rouge">L(A) < L(B)</code>, but <code class="highlighter-rouge">L(A) < L(B)</code> does not necessarily imply <code class="highlighter-rouge">A -> B</code>. In other words, <code class="highlighter-rouge">A -> B</code> implies that the logical clock of A is less than that of B, but the logical clock of A being less than that of B does <em>not</em> imply that <code class="highlighter-rouge">A -> B</code>.</p>
<h2 id="totally-ordered-multicast">Totally Ordered Multicast</h2>
<p><strong>Example</strong>: We have a large distributed database. We need to make sure that replications are seen in the same order in all replicas. This requires us to cast the replicas to all systems in an absolutely total order.</p>
<p><strong>Example Situation:</strong> The following events occur:</p>
<ul>
<li>A) We have $1000 in a bank account.</li>
<li>B) We add $100</li>
<li>C) We calculate 1% interest on the balance.</li>
</ul>
<p>If the order is ABC, then the 1% interest will be $1100 * 0.01 = $11. But if the order is ACB, then the interest will be $1000 * 0.01 = $10. In this case, the order matters as the interest is different.</p>
<p>Lamport’s logical clocks can be applied to implement a totally‐ordered multicast in a distributed system.</p>
<h3 id="implementation">Implementation</h3>
<p>Assumptions:</p>
<ul>
<li>No messages are lost</li>
<li>Messages from the same sender are received in the same order as they were sent</li>
</ul>
<p>Process <code class="highlighter-rouge">P(i)</code> will send out a message <code class="highlighter-rouge">M(i)</code> to all others with timestamp <code class="highlighter-rouge">T(i)</code>. An incoming message is queued according to it’s timestamp. <code class="highlighter-rouge">P(i)</code> will pass a message to its own application if it meets 2 criteria: the message is at the head of the queue, the message has been acked by all other processes.</p>
<!---
```diagram
sequenceDiagram
P(j)->>P(i): Puts Message m(j) at t=1
Note right of P(i): P(i) sends out m(i) at t=2 because the receipt of m(j) caused L(i) to increment by 1.
P(i)->>P(j): Puts Message m(i) at t=2
P(i)->>P(k): Puts Message m(i) at t=2
Note right of P(i): P(i) sends out m(i) to P(k) at t=2 before P(k) gets M(j) from P(j) at t=1. This is okay though because they are from different senders and the timestamps will sort it out.
Note left of P(j): Since all P(i..n) have ACKed M(i), we would normally be able to process it. However, M(i) is not at the head of the queue, M(j) is.
P(j)->>P(k): Puts Message m(j) at t=1
opt m(j) can be processed now
Note left of P(j): Since all P(i..n) have ACKed m(j) and m(j) is at the head of the queue, we can process it now.
P(i)->>P(i): Perform Message m(j)
P(j)->>P(j): Perform Message m(j)
P(k)->>P(k): Perform Message m(j)
end
opt M(i) can be processed now
Note left of P(j): Since all P(i..n) have ACKed m(i) and m(i) is at the head of the queue, we can process it now.
P(i)->>P(i): Perform Message m(i)
P(j)->>P(j): Perform Message m(i)
P(k)->>P(k): Perform Message m(i)
end
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/598942c362d82725a25fd056b83001b8.png" alt="diagram image" width="100%" /></p>
<p>All processes will end up with the same messages with the same timestamps, so order can be sorted out locally and therefore all messages are delivered in the same order.</p>Julian NadeauThis is a great presentation. Logical clocks are used to agree on order in which events occur. The absolute/real time is not important in this concept. Event ordering can be based on any number of factors. In a local system, CPU time can be used. But in a distributed system, there is no perfectly synchronized time or clock that can be used, and local times may not be in sync (and probably are not). Lamport suggested a logical clock be used to address this. Key concepts Processes exchange messages Message must be sent before received Send/receive used to order events and synchronize logical clocks Properties If A happens before B in the same process (or system), then A -> B A -> B also means that A sent the message and B means the receipt of it Relation is transitive: e.g A -> B and B -> C implies A -> C Unordered events are concurrent: A !-> B and B !-> A implies A || B Lamport’s Logical Clocks If A -> B then timestamp(A) < timestamp(B) Lamport’s Algorithm Note: A -> B implies L(A) < L(B), but L(A) < L(B) does not necessarily imply A -> B. In other words, A -> B implies that the logical clock of A is less than that of B, but the logical clock of A being less than that of B does not imply that A -> B. Totally Ordered Multicast Example: We have a large distributed database. We need to make sure that replications are seen in the same order in all replicas. This requires us to cast the replicas to all systems in an absolutely total order. Example Situation: The following events occur: A) We have $1000 in a bank account. B) We add $100 C) We calculate 1% interest on the balance. If the order is ABC, then the 1% interest will be $1100 * 0.01 = $11. But if the order is ACB, then the interest will be $1000 * 0.01 = $10. In this case, the order matters as the interest is different. Lamport’s logical clocks can be applied to implement a totally‐ordered multicast in a distributed system. Implementation Assumptions: No messages are lost Messages from the same sender are received in the same order as they were sent Process P(i) will send out a message M(i) to all others with timestamp T(i). An incoming message is queued according to it’s timestamp. P(i) will pass a message to its own application if it meets 2 criteria: the message is at the head of the queue, the message has been acked by all other processes. All processes will end up with the same messages with the same timestamps, so order can be sorted out locally and therefore all messages are delivered in the same order.bundler/lockfile_parser.rb2017-04-05T23:24:08+00:002017-04-05T23:24:08+00:00http://notes.jnadeau.ca/computers/2017/04/05/lockfile_parser<!---
```diagram
gantt
title file: /gems/bundler-1.14.6/lib/bundler/lockfile_parser.rb method: initialize
dateFormat s.SSS
"@platforms = []" :a1, 0.000, 0.001
"@sources = []" :a1, 0.001, 0.002
"@dependencies = []" :a1, 0.002, 0.003
"@state = nil" :a1, 0.003, 0.004
"@specs = {}" :a1, 0.004, 0.005
"@rubygems_aggregate = Source::Rubygems.new" :a1, 0.005, 0.006
"if lockfile.match(/<<<<<<<|=======|>>>>>>>|\|\|\|\|\|\|\|/)" :a1, 0.006, 0.007
"lockfile.split(/(?:\r?\n)+/).each do |line|" :a1, 0.007, 0.008
"if SOURCE.include?(line) (run 1445 times)" :a1, 0.008, 0.010
"@state = :source (run 72 times)" :a1, 0.010, 0.011
"parse_source(line) (run 72 times)" :a1, 0.011, 0.012
"elsif line == DEPENDENCIES (run 1373 times)" :a1, 0.012, 0.014
"elsif line == PLATFORMS (run 1372 times)" :a1, 0.014, 0.016
"elsif line == RUBY (run 1371 times)" :a1, 0.016, 0.018
"elsif line == BUNDLED (run 1371 times)" :a1, 0.018, 0.020
"elsif line =~ /^[^\s]/ (run 1370 times)" :a1, 0.020, 0.025
"elsif @state (run 1370 times)" :a1, 0.025, 0.027
"send('parse_{@state}', line) (run 1370 times)" :a1, 0.027, 0.077
"@state = :platform" :a1, 0.077, 0.078
"@state = :dependency" :a1, 0.078, 0.079
"@state = :bundled_with" :a1, 0.079, 0.080
"@sources << @rubygems_aggregate" :a1, 0.080, 0.081
"@specs = @specs.values.sort_by(&:identifier)" :a1, 0.081, 0.090
"warn_for_outdated_bundler_version" :a1, 0.090, 0.091
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/Screen Shot 2017-03-28 at 4.50.46 PM.png" alt="diagram image" width="100%" /></p>
<p>Here, we see that <code class="highlighter-rouge">parse_#{@state}</code> is the bulk of the work. This is a dynamic call to parse methods… is any one of them slower than another?</p>
<p>To solve this, I split out the dynamic line into a case statement to see which lines were being called.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elseif @state
<span class="gi">+ case @state.to_s
+ when 'source'
+ parse_source(line)
+ when 'dependency'
+ parse_dependency(line)
+ when 'spec'
+ parse_spec(line)
+ when 'platform'
+ parse_platform(line)
+ when 'bundled_with'
+ parse_bundled_with(line)
+ when 'ruby'
+ parse_ruby(line)
+ else
+ send("parse_#{@state}", line)
+ end
</span><span class="gd">- send("parse_#{@state}", line)
</span>end
</code></pre></div></div>
<p>By the diagram below, we can see the following from our case statement:</p>
<table>
<thead>
<tr>
<th>parse_state</th>
<th>number</th>
<th>time</th>
<th> </th>
</tr>
</thead>
<tbody>
<tr>
<td>parse_source</td>
<td>1131 times</td>
<td>32ms</td>
<td><code class="highlighter-rouge">SOURCE</code> did not include line, so it went to the case statement</td>
</tr>
<tr>
<td>parse_platform</td>
<td>1 time</td>
<td>1 ms</td>
<td>-</td>
</tr>
<tr>
<td>parse_dependency</td>
<td>237 times</td>
<td>15 ms</td>
<td>-</td>
</tr>
<tr>
<td>parse_bundled_with</td>
<td>1 time</td>
<td>1 ms</td>
<td>-</td>
</tr>
</tbody>
</table>
<!---
```diagram
gantt
title file: /gems/bundler-1.14.6/lib/bundler/lockfile_parser.rb method: initialize
dateFormat s.SSS
"@platforms = []" :a1, 0.000, 0.001
"@sources = []" :a1, 0.001, 0.002
"@dependencies = []" :a1, 0.002, 0.003
"@state = nil" :a1, 0.003, 0.004
"@specs = {}" :a1, 0.004, 0.005
"@rubygems_aggregate = Source::Rubygems.new" :a1, 0.005, 0.006
"if lockfile.match(/<<<<<<<|=======|>>>>>>>|\|\|\|\|\|\|\|/)" :a1, 0.006, 0.007
"lockfile.split(/(?:\r?\n)+/).each do |line|" :a1, 0.007, 0.008
"if SOURCE.include?(line) (run 1445 times)" :a1, 0.008, 0.012
"@state = :source (run 72 times)" :a1, 0.012, 0.013
"parse_source(line) (run 72 times)" :a1, 0.013, 0.014
"elsif line == DEPENDENCIES (run 1373 times)" :a1, 0.014, 0.015
"elsif line == PLATFORMS (run 1372 times)" :a1, 0.015, 0.017
"elsif line == RUBY (run 1371 times)" :a1, 0.017, 0.019
"elsif line == BUNDLED (run 1371 times)" :a1, 0.019, 0.021
"elsif line =~ /^[^\s]/ (run 1370 times)" :a1, 0.021, 0.024
"elsif @state (run 1370 times)" :a1, 0.024, 0.026
"case @state.to_s (run 1370 times)" :a1, 0.026, 0.029
"parse_source(line) (run 1131 times)" :a1, 0.029, 0.061
"@state = :platform" :a1, 0.061, 0.062
"parse_platform(line)" :a1, 0.062, 0.063
"@state = :dependency" :a1, 0.063, 0.064
"parse_dependency(line) (run 237 times)" :a1, 0.064, 0.079
"@state = :bundled_with" :a1, 0.079, 0.080
"parse_bundled_with(line)" :a1, 0.080, 0.081
"@sources << @rubygems_aggregate" :a1, 0.081, 0.082
"@specs = @specs.values.sort_by(&:identifier)" :a1, 0.082, 0.093
"warn_for_outdated_bundler_version" :a1, 0.093, 0.094
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/Screen Shot 2017-03-28 at 4.46.45 PM.png" alt="diagram image" width="100%" /></p>
<hr />
<h2 id="parse_source">parse_source</h2>
<!---
```diagram
gantt
title file: /gems/bundler-1.14.6/lib/bundler/lockfile_parser.rb method: parse_source
dateFormat s.SSS
"case line (run 1203 times)" :a1, 0.000, 0.003
"@current_source = nil (run 72 times)" :a1, 0.003, 0.004
"@opts = {} (run 72 times)" :a1, 0.004, 0.005
"@type = line (run 72 times)" :a1, 0.005, 0.006
"value = $2 (run 205 times)" :a1, 0.006, 0.007
"value = true if value == 'true' (run 205 times)" :a1, 0.007, 0.008
"value = false if value == 'false' (run 205 times)" :a1, 0.008, 0.009
"key = $1 (run 205 times)" :a1, 0.009, 0.010
"if @opts[key] (run 205 times)" :a1, 0.010, 0.011
"@opts[key] = value (run 205 times)" :a1, 0.011, 0.014
"case @type (run 72 times)" :a1, 0.014, 0.015
"@current_source = TYPES[@type].from_lock(@opts) (run 71 times)" :a1, 0.015, 0.016
"if @sources.include?(@current_source) (run 71 times)" :a1, 0.016, 0.017
"@sources << @current_source (run 71 times)" :a1, 0.017, 0.018
"parse_spec(line) (run 854 times)" :a1, 0.018, 0.057
"Array(@opts['remote']).each do |url|" :a1, 0.057, 0.058
"@rubygems_aggregate.add_remote(url)" :a1, 0.058, 0.059
"@current_source = @rubygems_aggregate" :a1, 0.059, 0.060
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/Screen Shot 2017-03-28 at 4.47.00 PM.png" alt="diagram image" width="100%" /></p>
<p><code class="highlighter-rouge">parse_spec</code> is the obvious bulk of this method, so let’s also look there.</p>
<hr />
<h2 id="parse_spec">parse_spec</h2>
<p>The parse spec code looks like so:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">parse_spec</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">line</span> <span class="o">=~</span> <span class="no">NAME_VERSION_4</span>
<span class="nb">name</span> <span class="o">=</span> <span class="vg">$1</span>
<span class="n">version</span> <span class="o">=</span> <span class="vg">$2</span>
<span class="n">platform</span> <span class="o">=</span> <span class="vg">$3</span>
<span class="n">version</span> <span class="o">=</span> <span class="no">Gem</span><span class="o">::</span><span class="no">Version</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">version</span><span class="p">)</span>
<span class="n">platform</span> <span class="o">=</span> <span class="n">platform</span> <span class="p">?</span> <span class="no">Gem</span><span class="o">::</span><span class="no">Platform</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">platform</span><span class="p">)</span> <span class="p">:</span> <span class="no">Gem</span><span class="o">::</span><span class="no">Platform</span><span class="o">::</span><span class="no">RUBY</span>
<span class="vi">@current_spec</span> <span class="o">=</span> <span class="no">LazySpecification</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">version</span><span class="p">,</span> <span class="n">platform</span><span class="p">)</span>
<span class="vi">@current_spec</span><span class="p">.</span><span class="nf">source</span> <span class="o">=</span> <span class="vi">@current_source</span>
<span class="c1"># Avoid introducing multiple copies of the same spec (caused by</span>
<span class="c1"># duplicate GIT sections)</span>
<span class="vi">@specs</span><span class="p">[</span><span class="vi">@current_spec</span><span class="p">.</span><span class="nf">identifier</span><span class="p">]</span> <span class="o">||=</span> <span class="vi">@current_spec</span>
<span class="k">elsif</span> <span class="n">line</span> <span class="o">=~</span> <span class="no">NAME_VERSION_6</span>
<span class="nb">name</span> <span class="o">=</span> <span class="vg">$1</span>
<span class="n">version</span> <span class="o">=</span> <span class="vg">$2</span>
<span class="n">version</span> <span class="o">=</span> <span class="n">version</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s2">","</span><span class="p">).</span><span class="nf">map</span><span class="p">(</span><span class="o">&</span><span class="ss">:strip</span><span class="p">)</span> <span class="k">if</span> <span class="n">version</span>
<span class="n">dep</span> <span class="o">=</span> <span class="no">Gem</span><span class="o">::</span><span class="no">Dependency</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">version</span><span class="p">)</span>
<span class="vi">@current_spec</span><span class="p">.</span><span class="nf">dependencies</span> <span class="o"><<</span> <span class="n">dep</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>It takes about 15-17ms to run all of it. I’d like to see how often each part is called.</p>
<ul>
<li>NAME_VERSION_4, called 374 times, took about 7ms</li>
<li>NAME_VERSION_6, called 480 times, took about 8ms</li>
</ul>
<p>Which means they take equally as long, but the NAME_VERSION_4 option is slower taking about 0.000044s for each run as opposed to 0.000035s for each run.</p>
<p>So, what is the difference between these two? Well NAME_VERSION_4 is a top level dependency, whereas NAME_VERSION_6 is a sub-dependency, it seems.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">4</span> <span class="n">web</span><span class="o">-</span><span class="n">console</span> <span class="p">(</span><span class="mf">3.4</span><span class="o">.</span><span class="mi">0</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">actionview</span> <span class="p">(</span><span class="o">>=</span> <span class="mf">5.0</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">activemodel</span> <span class="p">(</span><span class="o">>=</span> <span class="mf">5.0</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">debug_inspector</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">railties</span> <span class="p">(</span><span class="o">>=</span> <span class="mf">5.0</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">4</span> <span class="n">webmock</span> <span class="p">(</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">2</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">addressable</span> <span class="p">(</span><span class="o">>=</span> <span class="mf">2.3</span><span class="o">.</span><span class="mi">6</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">crack</span> <span class="p">(</span><span class="o">>=</span> <span class="mf">0.3</span><span class="o">.</span><span class="mi">2</span><span class="p">)</span>
<span class="no">NAME</span> <span class="no">VERSION</span> <span class="mi">6</span> <span class="n">hashdiff</span>
</code></pre></div></div>
<p>So what does this actually do? Seems it resolves specifications from the lockfile. The “4 space” (NAME VERSION 4) seems to also load a current spec, which I don’t quite get. Seems we re-assign this class level variable a lot to avoid passing it around.</p>
<!--
```diagram
gantt
title file: /gems/bundler-1.14.6/lib/bundler/lockfile_parser.rb method: parse_spec
dateFormat s.SSS
"if line =~ NAME_VERSION_4 (run 854 times)" :a1, 0.000, 0.004
"name = $1 (run 374 times)" :a1, 0.004, 0.005
"version = $2 (run 374 times)" :a1, 0.005, 0.006
"platform = $3 (run 374 times)" :a1, 0.006, 0.007
"version = Gem::Version.new(version) (run 374 times)" :a1, 0.007, 0.009
"platform = platform ? Gem::Platform.new(platform) : Gem::Platform::RUBY (run 374 times)" :a1, 0.009, 0.010
"@current_spec = LazySpecification.new(name version platform) (run 374 times)" :a1, 0.010, 0.013
"@current_spec.source = @current_source (run 374 times)" :a1, 0.013, 0.014
"elsif line =~ NAME_VERSION_6 (run 480 times)" :a1, 0.014, 0.016
"name = $1 (run 480 times)" :a1, 0.016, 0.018
"version = $2 (run 480 times)" :a1, 0.018, 0.019
"version = version.split(' ').map(&:strip) if version (run 480 times)" :a1, 0.019, 0.021
"dep = Gem::Dependency.new(name version) (run 480 times)" :a1, 0.021, 0.035
"@specs[@current_spec.identifier] ||= @current_spec" :a1, 0.035, 0.036
```
--->
<p><img src="https://jules2689.github.io/gitcdn/images/website/images/diagram/Screen Shot 2017-03-28 at 6.33.13 PM.png" alt="diagram image" width="100%" /></p>
<p>We can see that <code class="highlighter-rouge">"dep = GemDependency.new(name version) (run 480 times)" :a1, 0.021, 0.035</code> takes a chunk of time (14ms with gantt generation, 6ms in reality), otherwise there’s not much bulk here.</p>
<hr />
<p>So, in the end the reason this file is slower is that it is iterating over many sources and creating <code class="highlighter-rouge">Gem::Dependency</code> objects. There is likely something we could do to make <code class="highlighter-rouge">LockFileParser</code> faster, but the work likely won’t be worth the time spent.</p>
<p>There isn’t much we can do to make this file faster without caching using marshalling the data or something.</p>Julian NadeauHere, we see that parse_#{@state} is the bulk of the work. This is a dynamic call to parse methods… is any one of them slower than another? To solve this, I split out the dynamic line into a case statement to see which lines were being called. elseif @state + case @state.to_s + when 'source' + parse_source(line) + when 'dependency' + parse_dependency(line) + when 'spec' + parse_spec(line) + when 'platform' + parse_platform(line) + when 'bundled_with' + parse_bundled_with(line) + when 'ruby' + parse_ruby(line) + else + send("parse_#{@state}", line) + end - send("parse_#{@state}", line) end By the diagram below, we can see the following from our case statement: parse_state number time parse_source 1131 times 32ms SOURCE did not include line, so it went to the case statement parse_platform 1 time 1 ms - parse_dependency 237 times 15 ms - parse_bundled_with 1 time 1 ms - parse_source parse_spec is the obvious bulk of this method, so let’s also look there. parse_spec The parse spec code looks like so: def parse_spec(line) if line =~ NAME_VERSION_4 name = $1 version = $2 platform = $3 version = Gem::Version.new(version) platform = platform ? Gem::Platform.new(platform) : Gem::Platform::RUBY @current_spec = LazySpecification.new(name, version, platform) @current_spec.source = @current_source # Avoid introducing multiple copies of the same spec (caused by # duplicate GIT sections) @specs[@current_spec.identifier] ||= @current_spec elsif line =~ NAME_VERSION_6 name = $1 version = $2 version = version.split(",").map(&:strip) if version dep = Gem::Dependency.new(name, version) @current_spec.dependencies << dep end end It takes about 15-17ms to run all of it. I’d like to see how often each part is called. NAME_VERSION_4, called 374 times, took about 7ms NAME_VERSION_6, called 480 times, took about 8ms Which means they take equally as long, but the NAME_VERSION_4 option is slower taking about 0.000044s for each run as opposed to 0.000035s for each run. So, what is the difference between these two? Well NAME_VERSION_4 is a top level dependency, whereas NAME_VERSION_6 is a sub-dependency, it seems. NAME VERSION 4 web-console (3.4.0) NAME VERSION 6 actionview (>= 5.0) NAME VERSION 6 activemodel (>= 5.0) NAME VERSION 6 debug_inspector NAME VERSION 6 railties (>= 5.0) NAME VERSION 4 webmock (2.3.2) NAME VERSION 6 addressable (>= 2.3.6) NAME VERSION 6 crack (>= 0.3.2) NAME VERSION 6 hashdiff So what does this actually do? Seems it resolves specifications from the lockfile. The “4 space” (NAME VERSION 4) seems to also load a current spec, which I don’t quite get. Seems we re-assign this class level variable a lot to avoid passing it around. We can see that "dep = GemDependency.new(name version) (run 480 times)" :a1, 0.021, 0.035 takes a chunk of time (14ms with gantt generation, 6ms in reality), otherwise there’s not much bulk here. So, in the end the reason this file is slower is that it is iterating over many sources and creating Gem::Dependency objects. There is likely something we could do to make LockFileParser faster, but the work likely won’t be worth the time spent. There isn’t much we can do to make this file faster without caching using marshalling the data or something.