For The Best Thing In The World
2022-11-21T06:04:19+00:00
https://yonghaowu.github.io/
YongHaoHu
看 SICP 不如先看 The Little Schemer
2020-03-13T00:00:00+00:00
https://yonghaowu.github.io//2020/03/13/scheme
<p><img src="https://img-blog.csdnimg.cn/20210211125314723.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hpb2hpb2h1,size_16,color_FFFFFF,t_70#pic_center" alt="公众号" /></p>
<p>##函数式入门圣经——王垠力荐《The Little Schemer》</p>
<p>除了在知乎看到过一两次,首次正式得知《The Little Schemer》此书则是来自王垠的博客:</p>
<blockquote>
<p>Dan Friedman 是 Indiana 大学的教授,程序语言领域的创始人之一。他主要的著作《The Little Schemer》(前身叫《The Little Lisper》) 是程序语言界最具影响力的书籍之一。现在很多程序语言界的元老级人物,当年都是看这本 “小人书” 学会了 Lisp/Scheme,才决心进入这一领域。</p>
</blockquote>
<p>怼天怼地的王垠,在 <a href="http://www.yinwang.org/blog-cn/2012/07/04/dan-friedman">GTF - Great Teacher Friedman</a> 不遗余墨的表达了对 Dan Friedman 敬重与感激,文中满是对这位好老师的感激之情与知遇之恩。</p>
<p>恰逢我又在重新看 SICP,对,就是那本看起来不厚,习题多得要命的那本 <em>Structure and Interpretation of Computer Programs</em>(<a href="https://book.douban.com/subject/1148282/">计算机程序的构造和解释</a>)。当然这本书的赞誉满如繁星:<a href="https://www.zhihu.com/question/26549715/answer/34336593">https://www.zhihu.com/question/26549715/answer/34336593</a></p>
<p>过多的习题实在没有耐心,难以坚持,于是就先试试看《The Little Schemer》。</p>
<p>我在寒假中已把《The Little Schemer》看完,收获良多,如今重温一下,顺便写本书评。</p>
<p>相对于 SICP ,我更推荐各位先看《The Little Schemer》打打基础,当这是一个 tutorial,其一问一答式的写作方法会令你耳目一新的——-讲的更加循循善诱,鞭辟入里,而且没什么习题 - -。你可以很快就了解到怎么样写 scheme,递归的威力,以及了解怎么样写一个 scheme 解释器,顺带了解了 丘奇计数,y 组合子等。</p>
<p>我顺带在这里整理下这本书讨论了啥:</p>
<p>####玩具总动员</p>
<p>引入scheme 中基本元素<code class="language-plaintext highlighter-rouge">atom</code>(原子), <code class="language-plaintext highlighter-rouge">list</code>(列表), <code class="language-plaintext highlighter-rouge">car</code>(取列表的第一项),<code class="language-plaintext highlighter-rouge">cdr</code>(取列表除第一项的余下作为列表),<code class="language-plaintext highlighter-rouge">cons</code>(把 a 元素加到 b 列表中),<code class="language-plaintext highlighter-rouge"> null?</code>(判断是否为空) , <code class="language-plaintext highlighter-rouge">eq?</code>(是否相等)。</p>
<p>以上就是全部了,之后的所有东西就靠以上关键字实现,包括 sheme 解释器,y 组合子,删除列表第 x 项元素。</p>
<p>####处理,处理,反复处理。。。</p>
<p>引入函数<code class="language-plaintext highlighter-rouge">lambda</code>以及<code class="language-plaintext highlighter-rouge">or</code> 关键词(if lese 作用),引入递归概念,实现函数<code class="language-plaintext highlighter-rouge"> lat?</code>(判断列表里是否全为<code class="language-plaintext highlighter-rouge"> atom</code> 原子),<code class="language-plaintext highlighter-rouge">member?</code>(列表是否包含xx)等为例子。</p>
<h4 id="用-cons-构筑恢宏">用 cons 构筑恢宏</h4>
<p>通过实现 <code class="language-plaintext highlighter-rouge">rember</code>(删除列表某元素)引入 cons 构建/拼接列表,实现 <code class="language-plaintext highlighter-rouge">first</code>(取列表第一项),实现 <code class="language-plaintext highlighter-rouge">insertR</code>(在列表的某项后插入一个元素),<code class="language-plaintext highlighter-rouge">multiinsertR</code>(在列表的某项后插入一个列表内所有元素——听到这个用递归做是否就有点不习惯了呢)。</p>
<p>此章主要通过实现更多的函数,让读者更加熟悉递归实现函数的思维,以及如何写递归终止条件。</p>
<h4 id="数字游戏">数字游戏</h4>
<p>实现数字中的 <code class="language-plaintext highlighter-rouge">+</code>, <code class="language-plaintext highlighter-rouge">-</code>,<code class="language-plaintext highlighter-rouge">*</code>,<code class="language-plaintext highlighter-rouge">/</code>等方法,就是自己来做数字的这些功能,可能这样说对没有接触过丘奇计数的人有点奇怪,我举一个我在阿里校招中出过的一道面试题为例子:</p>
<p>以下C 语言程序的输出是什么?</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include <stdio.h>
</span><span class="kt">int</span> <span class="nf">lambda</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">a</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span><span class="k">else</span> <span class="p">{</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">a</span> <span class="o">-</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">b</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">return</span> <span class="n">lambda</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">mull_r</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">a</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span><span class="k">else</span> <span class="p">{</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">a</span> <span class="o">-</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">return</span> <span class="n">lambda</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">mull_r</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">))</span> <span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">88888</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">11111</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">lambda</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">));</span>
<span class="kt">int</span> <span class="n">c</span> <span class="o">=</span> <span class="mi">300</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">d</span> <span class="o">=</span> <span class="mi">400</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">mull_r</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">d</span><span class="p">));</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A. 77777 120400</p>
<p>B. 99999 120000</p>
<p>C. 99998 120100</p>
<p>D. 99999 119600</p>
<p>答案是 B,以上 C 语言就是简单的实现了 <code class="language-plaintext highlighter-rouge">+ , -</code> 功能(某种程度上)。</p>
<p>同理还实现了<code class="language-plaintext highlighter-rouge">=</code>, <code class="language-plaintext highlighter-rouge"><</code>以及<code class="language-plaintext highlighter-rouge">></code>,无非就是 a,b 同时递归-1,看谁先为 0之类。</p>
<p>接着就是实现 <code class="language-plaintext highlighter-rouge">len</code>(列表长度),<code class="language-plaintext highlighter-rouge"> all-nums</code>(提取列表中所有数字),<code class="language-plaintext highlighter-rouge">one?</code>(判断 n是否为 1)等。</p>
<h4 id="我的天都是星星">我的天!都是星星</h4>
<p>此章中重新实现以前实现过的函数的泛化版本(都在函数名后加一个*,所以说都是星星)。</p>
<p>比如<code class="language-plaintext highlighter-rouge">rember*</code>(这次接受的第二个参数不是原子了,是列表,列表中出现过的都要删掉);<code class="language-plaintext highlighter-rouge">insertR*</code>(同理)等, <code class="language-plaintext highlighter-rouge">eqlists</code>(判断两个列表是否全等)。就是参数都为列表了,让递归来的更猛烈一些。</p>
<h4 id="如影随形">如影随形</h4>
<p>引入算术表达式,如 <code class="language-plaintext highlighter-rouge">1+3</code>,<code class="language-plaintext highlighter-rouge">3*4+12</code> 等并写算术表达式解释器,算出结果。</p>
<p>另外,值得一提的是又提了一遍丘奇数,如:</p>
<blockquote>
<p>4 代表概念上的四。因为人们更习惯阿拉巴表示法,所以我们选择了这个符号。</p>
<p>但,(() () () ())也有同样效果,(I V)也可以。</p>
<p>我们可以用() 代表 0, 1就是 ( () ),2 就是(()())</p>
</blockquote>
<p>那么加法就可以用 cons 做列表拼接 <code class="language-plaintext highlighter-rouge">(cons (quote()) (quote()) )</code> 结果就是<code class="language-plaintext highlighter-rouge">(())</code>也就是 1。</p>
<p>作者最后用了一个函数lat来说明在做高级抽象时应该注意不适用的陷阱(阴影),也就是本章标题的含义。</p>
<h4 id="朋友及关系">朋友及关系</h4>
<p>写一个 <code class="language-plaintext highlighter-rouge">set?</code>函数(判断列表是否为 set也就是没有重复出现的元素),<code class="language-plaintext highlighter-rouge">makeset</code>(从列表中构建一个 set),<code class="language-plaintext highlighter-rouge">subset</code>(b 是否 a 的子集),<code class="language-plaintext highlighter-rouge">eqset?</code>, <code class="language-plaintext highlighter-rouge">interset?</code>等。</p>
<p>示例了如何抽象出一个子过程(函数),来增强代码的表达能力。</p>
<h4 id="lambda-终结者">lambda 终结者</h4>
<p>在把函数当做数据类型,作为参数传入函数使用时,引入 Curry-ing(柯里化)的概念:</p>
<div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">a</span><span class="p">)</span>
<span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">x</span><span class="p">)</span>
<span class="p">(</span><span class="nb">eq?</span> <span class="nv">x</span> <span class="nv">z</span><span class="p">))</span>
<span class="p">)</span>
</code></pre></div></div>
<p>如上,传入参数 apple 的时候,会返回函数</p>
<div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">apple</span><span class="p">)</span>
<span class="p">(</span><span class="nb">eq?</span> <span class="nv">apple</span> <span class="nv">z</span><span class="p">))</span>
</code></pre></div></div>
<p>如上就可以构造出一个函数,传给 <code class="language-plaintext highlighter-rouge">rember </code>函数(根据条件删除列表中元素)作为参数使用。</p>
<p>接着用这个抽象更高一层的函数,因为年代久远,我有些忘了。。这里描述不了了。</p>
<p>####。。。。周而复始。。。</p>
<p>这一章,作者从无到有的推导出在没有定义函数名字的时候,怎么样实现递归,也即是 Y conbinator(Y 组合子)的来由,然而实在让人头大,我看了好多遍,也只是似是而非,不能鞭辟入里的讲解出来,所以我算是不懂的。</p>
<h4 id="值是什么">值是什么</h4>
<p>有了递归,有了之前写过的数字表达式解释器,而 scheme 本来就很简单,于是这一章就可以总结之前学到的所有东西,写一个 scheme 解析器了。</p>
<hr />
<p>以上就是《The Little Schemer》的内容,对于一个刚入门学计算机的,没有接触过函数式编程的,我是极力推荐的。</p>
<p>努力学完理解完,一周时间勉强可以解决了,之后两章可能需要花比较长的时间去理解——难度暴涨。。。需要自己去多看看其他书了。</p>
<p>其实理解完除了最后两章的内容,上手 SICP 就非常简单了,只不过习题还需多加努力。</p>
源码剖析:如何写一个 redis driver 库驱动
2020-03-13T00:00:00+00:00
https://yonghaowu.github.io//2020/03/13/redis
<p><img src="https://img-blog.csdnimg.cn/20210211125314723.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hpb2hpb2h1,size_16,color_FFFFFF,t_70#pic_center" alt="公众号" /></p>
<p>from : https://mp.weixin.qq.com/s/D_xlof0mNs4UZi973vaOXw 有些图裂了,看原文比较方便~</p>
<p>###前言</p>
<p>最近跟同事请教了一下 redis 相关的事情,就找来了一下 redis 的驱动,看看这些库是怎么做 <code class="language-plaintext highlighter-rouge">redis cluster</code>的 <code class="language-plaintext highlighter-rouge">pipeline</code> 以及 <code class="language-plaintext highlighter-rouge">transaction</code>的,以下就把相关流程的代码剖析一下,还是有一些有意思的点的。</p>
<p>因为 C 语言比较底层,其他语言感觉描述性都差了一点,我找的是 elixir 的库来看的,质量很高。</p>
<p>事后才发现原来这个elixir 的 redis 库的作者是 elixir 这门语言的核心开发者;P</p>
<p>正文开始。</p>
<p>首先呢, Elixir 的这个库不支持 redis 集群,后来有人基于它扩展成支持简单的集群,所以先讲普通的怎么做,再扩展。</p>
<h3 id="架构">架构</h3>
<p>这个库是单进程异步,当你发命令过来时,此库处理完后会马上发给 Redis 服务器,然后就可以接收新的命令,当 Redis Server 答复时,会返回此<code class="language-plaintext highlighter-rouge">Reply</code>给你。</p>
<p>一般连接池有通用的库,所以交给调用方来做,库只处理每个连接的请求。</p>
<h4 id="resp-redis-serialization-protocol">RESP (REdis Serialization Protocol)</h4>
<p>ps,上面这个标题就是来自 redis 官网的,明显 <code class="language-plaintext highlighter-rouge">RE</code>是 typo。</p>
<p>Redis 用的协议<code class="language-plaintext highlighter-rouge">RESP</code>是自己定的文本协议,客户端与服务端直接通过 TCP 连接通讯。</p>
<p>这个文本协议,其实就是对数据的序列化,以下就是规则:</p>
<ul>
<li>For <strong>Simple Strings</strong> the first byte of the reply is “+”</li>
<li>For <strong>Errors</strong> the first byte of the reply is “-“</li>
<li>For <strong>Integers</strong> the first byte of the reply is “:”</li>
<li>For <strong>Bulk Strings</strong> the first byte of the reply is “$”</li>
<li>For <strong>Arrays</strong> the first byte of the reply is “<code class="language-plaintext highlighter-rouge">*</code>”</li>
</ul>
<p>对于客户端而言,发过去给服务器的命令其实数据结构就是数组,所以只需要<code class="language-plaintext highlighter-rouge">*数组长度\r\n$数组[0]里命令的长度\r\n 数组[0]里命令</code>。</p>
<p>说起来有点抽象,看看实际例子:</p>
<ul>
<li>
<p>` LLEN mylist<code class="language-plaintext highlighter-rouge"> 按照协议 encode就变成 </code>*2\r\n$4\r\nLLEN\r\n$6\r\nmylist\r\n` 的文本,</p>
<ul>
<li>数组里有两个字符串,分别是 4 长度的<code class="language-plaintext highlighter-rouge">LLEN</code>以及 6 个字符的<code class="language-plaintext highlighter-rouge">mylist</code></li>
</ul>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">SET mykey 1</code>按协议 encode 就变成<code class="language-plaintext highlighter-rouge">*3\r\n$3\r\nSET\r\n$5\r\nmykey\r\n$1\r\n1\r\n"</code></p>
<ul>
<li>数组里有三个字符串,分别是 3 长度的<code class="language-plaintext highlighter-rouge">SET</code>以及 5 个字符的<code class="language-plaintext highlighter-rouge">mykey</code>,还有 1 个字符的<code class="language-plaintext highlighter-rouge">1</code></li>
</ul>
<p>可以看看这个库是怎么做的,就是递归拼接,记录数组的长度,最后在最开头拼上<code class="language-plaintext highlighter-rouge">*数组长度</code>。</p>
</li>
</ul>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">@doc</span> <span class="sx">~S""</span><span class="s2">"
Packs a list of Elixir terms to a Redis (RESP) array.
This function returns an iodata (instead of a binary) because the packed
result is usually sent to Redis through `:gen_tcp.send/2` or similar. It can
be converted to a binary with `IO.iodata_to_binary/1`.
All elements of `elems` are converted to strings with `to_string/1`, hence
this function supports encoding everything that implements `String.Chars`.
## Examples
iex> iodata = Redix.Protocol.pack(["</span><span class="no">SET</span><span class="s2">", "</span><span class="n">mykey</span><span class="s2">", 1])
iex> IO.iodata_to_binary(iodata)
"</span><span class="o">*</span><span class="mi">3</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">n</span><span class="err">$</span><span class="mi">3</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">nSET</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">n</span><span class="err">$</span><span class="mi">5</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">nmykey</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">n</span><span class="err">$</span><span class="mi">1</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">n1</span><span class="p">\</span><span class="n">r</span><span class="p">\</span><span class="n">n</span><span class="s2">"
"""</span>
<span class="nv">@crlf_iodata</span> <span class="p">[</span><span class="err">?</span><span class="p">\</span><span class="n">r</span><span class="p">,</span> <span class="err">?</span><span class="p">\</span><span class="n">n</span><span class="p">]</span>
<span class="nv">@spec</span> <span class="n">pack</span><span class="p">([</span><span class="n">binary</span><span class="p">])</span> <span class="p">::</span> <span class="n">iodata</span>
<span class="k">def</span> <span class="n">pack</span><span class="p">(</span><span class="n">items</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_list</span><span class="p">(</span><span class="n">items</span><span class="p">)</span> <span class="k">do</span>
<span class="n">pack</span><span class="p">(</span><span class="n">items</span><span class="p">,</span> <span class="p">[],</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">pack</span><span class="p">([</span><span class="n">item</span> <span class="o">|</span> <span class="n">rest</span><span class="p">],</span> <span class="n">acc</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="k">do</span>
<span class="n">item</span> <span class="o">=</span> <span class="n">to_string</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
<span class="n">new_acc</span> <span class="o">=</span> <span class="p">[</span><span class="n">acc</span><span class="p">,</span> <span class="p">[</span><span class="sx">?$</span><span class="p">,</span> <span class="no">Integer</span><span class="o">.</span><span class="n">to_string</span><span class="p">(</span><span class="n">byte_size</span><span class="p">(</span><span class="n">item</span><span class="p">)),</span> <span class="nv">@crlf_iodata</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="nv">@crlf_iodata</span><span class="p">]]</span>
<span class="n">pack</span><span class="p">(</span><span class="n">rest</span><span class="p">,</span> <span class="n">new_acc</span><span class="p">,</span> <span class="n">count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">pack</span><span class="p">([],</span> <span class="n">acc</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="k">do</span>
<span class="p">[</span><span class="sx">?*</span><span class="p">,</span> <span class="no">Integer</span><span class="o">.</span><span class="n">to_string</span><span class="p">(</span><span class="n">count</span><span class="p">),</span> <span class="nv">@crlf_iodata</span><span class="p">,</span> <span class="n">acc</span><span class="p">]</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="维护长连接">维护长连接</h3>
<p>作为 client 的库,维护长连接,避免频繁创建连接,这个是常规操作。</p>
<p>而有趣的是,作者使用了 <code class="language-plaintext highlighter-rouge">erlang OTP</code>自带的状态机框架 <code class="language-plaintext highlighter-rouge">gen_statem</code> 来维持 TCP 长连接,这个功能是<code class="language-plaintext highlighter-rouge">OTP 19</code>也就是 16 年才推出的,在不知道此作者是 elixir 语言的贡献者前,我还小小的膜拜了一下。</p>
<p>状态机如下图,初始状态不是同步连接,就是connecting 状态;同步的话,成功就是处于 connected 状态。</p>
<p>状态的动作依靠 <code class="language-plaintext highlighter-rouge">TCP</code> 的事件消息来驱动,状态转移自己控制。</p>
<p><img src="assets/image-20191020125257990.png" alt="image-20191020125257990" /></p>
<p>举例子:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">disconnected</span><span class="p">({</span><span class="ss">:timeout</span><span class="p">,</span> <span class="ss">:reconnect</span><span class="p">},</span> <span class="n">_timer_info</span><span class="p">,</span> <span class="p">%</span><span class="bp">__MODULE__</span><span class="p">{}</span> <span class="o">=</span> <span class="n">data</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">socket_owner</span><span class="p">}</span> <span class="o">=</span> <span class="no">SocketOwner</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">self</span><span class="p">(),</span> <span class="n">data</span><span class="o">.</span><span class="n">opts</span><span class="p">,</span> <span class="n">data</span><span class="o">.</span><span class="n">table</span><span class="p">)</span>
<span class="n">new_data</span> <span class="o">=</span> <span class="p">%{</span><span class="n">data</span> <span class="o">|</span> <span class="ss">socket_owner:</span> <span class="n">socket_owner</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:next_state</span><span class="p">,</span> <span class="ss">:connecting</span><span class="p">,</span> <span class="n">new_data</span><span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>以上代码就是在 <code class="language-plaintext highlighter-rouge">discconected</code>状态收到 <code class="language-plaintext highlighter-rouge">TCP</code> 的<code class="language-plaintext highlighter-rouge">{:timeout, :reconnect}</code>消息,创建一个新的<code class="language-plaintext highlighter-rouge">TCP socket</code>进程,将状态转移到<code class="language-plaintext highlighter-rouge">:connecting</code>。</p>
<p>而 <code class="language-plaintext highlighter-rouge">socket</code> 进程在初始化时,会发送<code class="language-plaintext highlighter-rouge">connect</code>消息给自己:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">handle_info</span><span class="p">(</span><span class="ss">:connect</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="k">do</span>
<span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">socket</span><span class="p">,</span> <span class="n">address</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Connector</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">opts</span><span class="p">),</span>
<span class="ss">:ok</span> <span class="o"><-</span> <span class="n">setopts</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">socket</span><span class="p">,</span> <span class="ss">active:</span> <span class="ss">:once</span><span class="p">)</span> <span class="k">do</span>
<span class="n">send</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">conn</span><span class="p">,</span> <span class="p">{</span><span class="ss">:connected</span><span class="p">,</span> <span class="n">self</span><span class="p">(),</span> <span class="n">socket</span><span class="p">,</span> <span class="n">address</span><span class="p">})</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">%{</span><span class="n">state</span> <span class="o">|</span> <span class="ss">socket:</span> <span class="n">socket</span><span class="p">}}</span>
<span class="k">else</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">reason</span><span class="p">}</span> <span class="o">-></span> <span class="n">stop</span><span class="p">(</span><span class="n">reason</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:stop</span><span class="p">,</span> <span class="n">reason</span><span class="p">}</span> <span class="o">-></span> <span class="n">stop</span><span class="p">(</span><span class="n">reason</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>成功了,就发送<code class="language-plaintext highlighter-rouge">connected</code>消息给原来的<code class="language-plaintext highlighter-rouge"> 状态机进程(也就是 connection 进程)</code>,<code class="language-plaintext highlighter-rouge">connection</code>进程处于<code class="language-plaintext highlighter-rouge">connecting</code>状态时,接受此消息,更新 socket 信息,状态转移到 <code class="language-plaintext highlighter-rouge">connected</code>。</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">connecting</span><span class="p">(</span>
<span class="ss">:info</span><span class="p">,</span>
<span class="p">{</span><span class="ss">:connected</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">socket</span><span class="p">,</span> <span class="n">address</span><span class="p">},</span>
<span class="p">%</span><span class="bp">__MODULE__</span><span class="p">{</span><span class="ss">socket_owner:</span> <span class="n">owner</span><span class="p">}</span> <span class="o">=</span> <span class="n">data</span>
<span class="p">)</span> <span class="k">do</span>
<span class="k">if</span> <span class="n">data</span><span class="o">.</span><span class="n">backoff_current</span> <span class="k">do</span>
<span class="ss">:telemetry</span><span class="o">.</span><span class="n">execute</span><span class="p">([</span><span class="ss">:redix</span><span class="p">,</span> <span class="ss">:reconnection</span><span class="p">],</span> <span class="p">%{},</span> <span class="p">%{</span>
<span class="ss">connection:</span> <span class="n">data</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="ss">:name</span><span class="p">]</span> <span class="o">||</span> <span class="n">self</span><span class="p">(),</span>
<span class="ss">address:</span> <span class="n">address</span>
<span class="p">})</span>
<span class="k">end</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">%{</span><span class="n">data</span> <span class="o">|</span> <span class="ss">socket:</span> <span class="n">socket</span><span class="p">,</span> <span class="ss">backoff_current:</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">connected_address:</span> <span class="n">address</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:next_state</span><span class="p">,</span> <span class="ss">:connected</span><span class="p">,</span> <span class="p">%{</span><span class="n">data</span> <span class="o">|</span> <span class="ss">socket:</span> <span class="n">socket</span><span class="p">}}</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="执行命令">执行命令</h3>
<p>Redis 执行命令主要有 <code class="language-plaintext highlighter-rouge">Comand</code>、<code class="language-plaintext highlighter-rouge">Pipeline</code>以及<code class="language-plaintext highlighter-rouge">Trasaction</code>三种概念:</p>
<ul>
<li>` command`:一问一答式的,客户端等待 server 返回消息;</li>
<li><code class="language-plaintext highlighter-rouge">Pipeline</code>:发送一连串命令,这些命令发往 server,不用一问一答,收到命令马上返回。sever 以队列执行,执行完后全部结果返回回来;</li>
<li><code class="language-plaintext highlighter-rouge">Trasaction</code>:依靠<code class="language-plaintext highlighter-rouge">MULTI</code>/<code class="language-plaintext highlighter-rouge">EXEC</code>命令,<code class="language-plaintext highlighter-rouge">MULTI</code>命令开始<code class="language-plaintext highlighter-rouge">Trasaction</code>,此后发送的命令都存到 server 的队列里,<code class="language-plaintext highlighter-rouge">EXEC</code>命令发送后马上这队列里所有命令;期间不会有其他命令影响这些命令的执行。</li>
</ul>
<p>库里把 <code class="language-plaintext highlighter-rouge">Command</code> 命令用 <code class="language-plaintext highlighter-rouge">Pipeline</code>来做,其实本质是一样的。</p>
<h5 id="pipeline">Pipeline</h5>
<p>以下的<code class="language-plaintext highlighter-rouge">pipeline</code>就是负责用户调用的函数,<code class="language-plaintext highlighter-rouge">:gen_statem.cast</code>就是把消息数据传给状态机,接着就是起了一个进程来监控这个连接,挂了就退出;同时阻塞等待状态机完成处理获得数据后发消息过来。</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">pipeline</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">commands</span><span class="p">,</span> <span class="n">timeout</span><span class="p">)</span> <span class="k">do</span>
<span class="n">conn</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">whereis</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span>
<span class="n">request_id</span> <span class="o">=</span> <span class="no">Process</span><span class="o">.</span><span class="n">monitor</span><span class="p">(</span><span class="n">conn</span><span class="p">)</span>
<span class="c1"># We cast to the connection process knowing that it will reply at some point,</span>
<span class="c1"># either after roughly timeout or when a response is ready.</span>
<span class="n">cast</span> <span class="o">=</span> <span class="p">{</span><span class="ss">:pipeline</span><span class="p">,</span> <span class="n">commands</span><span class="p">,</span> <span class="n">_from</span> <span class="o">=</span> <span class="p">{</span><span class="n">self</span><span class="p">(),</span> <span class="n">request_id</span><span class="p">},</span> <span class="n">timeout</span><span class="p">}</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="ss">:gen_statem</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">cast</span><span class="p">)</span>
<span class="k">receive</span> <span class="k">do</span>
<span class="p">{</span><span class="o">^</span><span class="n">request_id</span><span class="p">,</span> <span class="n">resp</span><span class="p">}</span> <span class="o">-></span>
<span class="n">_</span> <span class="o">=</span> <span class="no">Process</span><span class="o">.</span><span class="n">demonitor</span><span class="p">(</span><span class="n">request_id</span><span class="p">,</span> <span class="p">[</span><span class="ss">:flush</span><span class="p">])</span>
<span class="n">resp</span>
<span class="p">{</span><span class="ss">:DOWN</span><span class="p">,</span> <span class="o">^</span><span class="n">request_id</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">reason</span><span class="p">}</span> <span class="o">-></span>
<span class="k">exit</span><span class="p">(</span><span class="n">reason</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>状态机这块的代码就是:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">connected</span><span class="p">(</span><span class="ss">:cast</span><span class="p">,</span> <span class="p">{</span><span class="ss">:pipeline</span><span class="p">,</span> <span class="n">commands</span><span class="p">,</span> <span class="n">from</span><span class="p">,</span> <span class="n">timeout</span><span class="p">},</span> <span class="n">data</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="n">ncommands</span><span class="p">,</span> <span class="n">data</span><span class="p">}</span> <span class="o">=</span> <span class="n">get_client_reply</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">commands</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ncommands</span> <span class="o">></span> <span class="mi">0</span> <span class="k">do</span>
<span class="p">{</span><span class="n">counter</span><span class="p">,</span> <span class="n">data</span><span class="p">}</span> <span class="o">=</span> <span class="n">get_and_update_in</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">counter</span><span class="p">,</span> <span class="o">&</span><span class="p">{</span><span class="nv">&1</span><span class="p">,</span> <span class="nv">&1</span> <span class="o">+</span> <span class="mi">1</span><span class="p">})</span>
<span class="n">row</span> <span class="o">=</span> <span class="p">{</span><span class="n">counter</span><span class="p">,</span> <span class="n">from</span><span class="p">,</span> <span class="n">ncommands</span><span class="p">,</span> <span class="n">_timed_out?</span> <span class="o">=</span> <span class="no">false</span><span class="p">}</span>
<span class="ss">:ets</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">table</span><span class="p">,</span> <span class="n">row</span><span class="p">)</span>
<span class="k">case</span> <span class="n">data</span><span class="o">.</span><span class="n">transport</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">socket</span><span class="p">,</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">commands</span><span class="p">,</span> <span class="o">&</span><span class="no">Protocol</span><span class="o">.</span><span class="n">pack</span><span class="o">/</span><span class="mi">1</span><span class="p">))</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">-></span>
<span class="n">actions</span> <span class="o">=</span>
<span class="k">case</span> <span class="n">timeout</span> <span class="k">do</span>
<span class="ss">:infinity</span> <span class="o">-></span> <span class="p">[]</span>
<span class="n">_other</span> <span class="o">-></span> <span class="c1">#[\{\{:timeout, {:client_timed_out, counter\}\}, timeout, from}]</span>
<span class="k">end</span>
<span class="p">{</span><span class="ss">:keep_state</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">actions</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">_reason</span><span class="p">}</span> <span class="o">-></span>
<span class="c1"># The socket owner will get a closed message at some point, so we just move to the</span>
<span class="c1"># disconnected state.</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">transport</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">socket</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:next_state</span><span class="p">,</span> <span class="ss">:disconnected</span><span class="p">,</span> <span class="n">data</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">else</span>
<span class="n">reply</span><span class="p">(</span><span class="n">from</span><span class="p">,</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">[]})</span>
<span class="p">{</span><span class="ss">:keep_state</span><span class="p">,</span> <span class="n">data</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>没什么特别的,<code class="language-plaintext highlighter-rouge">get_client_reply</code>就是处理客户端是否想得到服务器回复的命令的 <code class="language-plaintext highlighter-rouge">CLIENT REPLY</code>的各种指令,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> defp get_client_reply([command | rest], ncommands, client_reply) do
case parse_client_reply(command) do
:off -> get_client_reply(rest, ncommands, :off)
:skip when client_reply == :off -> get_client_reply(rest, ncommands, :off)
:skip -> get_client_reply(rest, ncommands, :skip)
:on -> get_client_reply(rest, ncommands + 1, :on)
nil when client_reply == :on -> get_client_reply(rest, ncommands + 1, client_reply)
nil when client_reply == :off -> get_client_reply(rest, ncommands, client_reply)
nil when client_reply == :skip -> get_client_reply(rest, ncommands, :on)
end
end
</code></pre></div></div>
<p>接着就是把命令序列号成 RESP,使用<code class="language-plaintext highlighter-rouge">data.transport.send</code>发送给服务器,其实 Redis 除了 TCP 外还可以使用SSL/TLS 协议,所以就有了这一层抽象。</p>
<p>如果是 TCP,那么socket 服务就会在 redis 服务器返回消息后,此函数接收自动处理:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def handle_info({transport, socket, data}, %__MODULE__{socket: socket} = state)
when transport in [:tcp, :ssl] do
:ok = setopts(state, socket, active: :once)
state = new_data(state, data)
{:noreply, state}
end
</code></pre></div></div>
<h2 id="支持redis-cluster">支持Redis Cluster</h2>
<h4 id="redis-cluster-的分布式算法">Redis Cluster 的分布式算法</h4>
<p>官网写的很好了,我简单说一下好了。</p>
<blockquote>
<p>Redis Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call an <strong>hash slot</strong>.</p>
</blockquote>
<p><code class="language-plaintext highlighter-rouge">Redis Cluster</code>没有用一致性哈希算法,而是用了<code class="language-plaintext highlighter-rouge">hash slot</code>(哈希桶)</p>
<blockquote>
<p>There are 16384 hash slots in Redis Cluster, and to compute what is the hash slot of a given key, we simply take the CRC16 of the key modulo 16384.</p>
</blockquote>
<p>redis 会固定分配 16384 个 slots 到不同的节点,用的算法就是对 key 做 CRC16 然后对 16384取模: <code class="language-plaintext highlighter-rouge">HASH_SLOT = CRC16(key) mod 16384</code></p>
<p>例子如下:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Every node in a Redis Cluster is responsible for a subset of the hash slots, so for example you may have a cluster with 3 nodes, where:
- Node A contains hash slots from 0 to 5500.
- Node B contains hash slots from 5501 to 11000.
- Node C contains hash slots from 11001 to 16383.
</code></pre></div></div>
<blockquote>
<p>This allows to add and remove nodes in the cluster easily. For example if I want to add a new node D, I need to move some hash slot from nodes A, B, C to D. Similarly if I want to remove node A from the cluster I can just move the hash slots served by A to B and C. When the node A will be empty I can remove it from the cluster completely.</p>
</blockquote>
<p>用这样的算法,比一致性哈希方便,更有操作性:</p>
<blockquote>
<p>Redis Cluster implements a concept called <strong>hash tags</strong> that can be used in order to force certain keys to be stored in the same hash slot.</p>
<p>Because moving hash slots from a node to another does not require to stop operations, adding and removing nodes, or changing the percentage of hash slots hold by nodes, does not require any downtime.</p>
</blockquote>
<p>对于 redis 或者对用户来说,可以轻松地分配移动 slots;</p>
<p>而一致性哈希就只能自己算虚拟节点,并且『祈求』之后请求量多了最终达到想要的平衡了。</p>
<p><img src="assets/image-20191020181958699.png" alt="image-20191020181958699" /></p>
<p>#####redix-cluster</p>
<p>原版没有支持集群,<a href="https://github.com/zhongwencool">zhongwencool</a>/<strong>redix-cluster</strong> 写了一个简单的包装版本。</p>
<p>只需要看这段,就很清楚为了集群做了些啥:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">@spec</span> <span class="n">pipeline</span><span class="p">([</span><span class="n">command</span><span class="p">],</span> <span class="no">Keyword</span><span class="o">.</span><span class="n">t</span><span class="p">)</span> <span class="p">::</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">term</span><span class="p">}</span> <span class="o">|</span><span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">term</span><span class="p">}</span>
<span class="k">def</span> <span class="n">pipeline</span><span class="p">(</span><span class="n">pipeline</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">RedixCluster</span><span class="o">.</span><span class="no">Monitor</span><span class="o">.</span><span class="n">get_slot_cache</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:cluster</span><span class="p">,</span> <span class="n">slots_maps</span><span class="p">,</span> <span class="n">slots</span><span class="p">,</span> <span class="n">version</span><span class="p">}</span> <span class="o">-></span>
<span class="n">pipeline</span>
<span class="o">|></span> <span class="n">parse_keys_from_pipeline</span>
<span class="o">|></span> <span class="n">keys_to_slot_hashs</span>
<span class="o">|></span> <span class="n">is_same_slot_hashs</span>
<span class="o">|></span> <span class="n">get_pool_by_slot</span><span class="p">(</span><span class="n">slots_maps</span><span class="p">,</span> <span class="n">slots</span><span class="p">,</span> <span class="n">version</span><span class="p">)</span>
<span class="o">|></span> <span class="n">query_redis_pool</span><span class="p">(</span><span class="n">pipeline</span><span class="p">,</span> <span class="ss">:pipeline</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:not_cluster</span><span class="p">,</span> <span class="n">version</span><span class="p">,</span> <span class="n">pool_name</span><span class="p">}</span> <span class="o">-></span>
<span class="n">query_redis_pool</span><span class="p">({</span><span class="n">version</span><span class="p">,</span> <span class="n">pool_name</span><span class="p">},</span> <span class="n">pipeline</span><span class="p">,</span> <span class="ss">:pipeline</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">|></code> 就是类似 unix 的 <code class="language-plaintext highlighter-rouge">管道 |</code>,把函数返回值当做下个函数的第一个参数传给他。</p>
<p><code class="language-plaintext highlighter-rouge">get_slot_cache</code>就是获取redis的<code class="language-plaintext highlighter-rouge">cluster slots</code>这个记录,并且缓存起来。</p>
<blockquote>
<p><a href="https://redis.io/commands/cluster-slots">CLUSTER SLOTS</a> returns details about which cluster slots map to which Redis instances.</p>
</blockquote>
<ul>
<li><code class="language-plaintext highlighter-rouge">parse_keys_from_pipeline</code> 将全部 keys 从<code class="language-plaintext highlighter-rouge">Pineline</code> 命令里提取出来</li>
<li><code class="language-plaintext highlighter-rouge">keys_to_slot_hashs</code> 找出 各个key 在哪个 hash slot</li>
<li><code class="language-plaintext highlighter-rouge">is_same_slot_hashs</code> 判断所有 key 是不是在同一个 hash slot,是的,这个还不支持跨 slot,我在准备帮他写一个</li>
<li><code class="language-plaintext highlighter-rouge">get_pool_by_slot</code> 项目用了连接池来管理,所以要根据名字找对应的连接</li>
<li><code class="language-plaintext highlighter-rouge">query_redis_pool</code> 就是调用 原来的 Redix 做处理了</li>
</ul>
<p>简单来说,这个库就是残废的,哈哈哈。。。</p>
<p>不支持分布不同 slot,就是玩具。</p>
<p><img src="assets/image-20191020232650627.png" alt="image-20191020232650627" style="zoom: 50%;" /></p>
<h3 id="后文">后文</h3>
<p>总的来说就是这样子,还算是有挺多有趣的地方的。</p>
<p>欢迎转发,关注量这么少的我,越来越不想在公众号发文章了。。┑( ̄Д  ̄)┍</p>
实现一个linux初版的git
2020-03-13T00:00:00+00:00
https://yonghaowu.github.io//2020/03/13/implementGit
<p><img src="https://img-blog.csdnimg.cn/20210211125314723.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hpb2hpb2h1,size_16,color_FFFFFF,t_70#pic_center" alt="公众号" /></p>
<p>[TOC]</p>
<h1 id="naive-git">Naive Git</h1>
<h2 id="前言">前言</h2>
<p>我与两个师弟一起成立一个<code class="language-plaintext highlighter-rouge">git org</code>,主要是他们(我需要工作,划水出主意做PM居多)做一些趣味使然的项目,<a href="https://github.com/PioneerIncubator">PioneerIncubator</a>,这个<code class="language-plaintext highlighter-rouge">git</code>是第三个项目,第一个项目是<code class="language-plaintext highlighter-rouge">betterGo</code>,我好几个月前就写好初版了,就等他们做一些完善补充工作了,之后会单独介绍。第二个项目是刚动手,他们搜了一下,发现上年十月发现有人做了,那个项目还有500多star了。</p>
<h2 id="git的原理是怎么样呢">Git的原理是怎么样呢?</h2>
<blockquote>
<p>Git is a distributed version-control system for tracking changes in source code during software development.</p>
</blockquote>
<p>各位读者就算不了解git的原理,想必也会用三把斧<code class="language-plaintext highlighter-rouge">git add; git commit; git push</code>,下面就简单说一下git是怎么做的版本管理的:跟踪文件的变化,使用commit作为标记,与远程服务器同步。</p>
<h3 id="跟踪文件变化">跟踪文件变化</h3>
<p>假如你来开发git这个工具,在初始化一个文件夹(repository)后,为了记录之后可能的修改,你需要记录当前所有需要跟踪的文件内容,最简单的就是全部复制一份好了。</p>
<p>文件是否变化了?比较一下文件哈希好了。</p>
<h3 id="commit作标记">Commit作标记</h3>
<p>顾言思义,就是将当前的<code class="language-plaintext highlighter-rouge">repository</code>状态存储起来,作为commit。你可以通过<code class="language-plaintext highlighter-rouge">commit</code>恢复到任意状态,<code class="language-plaintext highlighter-rouge">git tag</code>本质也只是给这个<code class="language-plaintext highlighter-rouge">commit</code>一个<code class="language-plaintext highlighter-rouge">tag</code>(别名),<code class="language-plaintext highlighter-rouge">git branch</code> 也是一样。</p>
<p>恢复到某一个<code class="language-plaintext highlighter-rouge">commit</code>,就是将它所代表的<code class="language-plaintext highlighter-rouge">repository</code>状态恢复起来,就是将文件全部内容以及当前commit恢复到那个状态。</p>
<h3 id="与远程服务器同步">与远程服务器同步</h3>
<p>git说自己是分布式的版本管理系统,是因为假如A、B、C三个人一起合作,理论上每个人都有一份server的版本,而且可以独立开发,解决冲突。</p>
<p>####</p>
<h2 id="git具体是怎么做的呢">Git具体是怎么做的呢?</h2>
<p>原理说完了,但commit的管理是要用东西来存储读取管理的,Git没有用数据库,直接将其内容放到<code class="language-plaintext highlighter-rouge">.git</code>文件夹里。</p>
<p>里面有什么内容呢?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> .
|-- HEAD //指向branch、tag (ref: refs/heads/devbranch)
|-- index
|-- objects
| |-- 05
| | `-- 76fac355dd17e39fd2671b010e36299f713b4d
| |-- 0c
| | `-- 819c497e4eca8e08422e61adec781cc91d125d
| |-- fe
| | `-- 897108953cc224f417551031beacc396b11fb0
| |-- fe
| | `-- 897108953cc224f417551031beacc396b11fb0
| |-- info
|
`-- refs
|-- heads //各个branch的heads
| `-- master //此分支最新的commit id
| `-- devBranch // checkout -b branch就会生成的branch
`-- tags
`-- v0.1
</code></pre></div></div>
<p>各位再结合</p>
<p>下面我展开讲讲:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">HEAD</code>: 指向branch或者tag,标记当前是在哪个分支或者tag上;</li>
<li><code class="language-plaintext highlighter-rouge">index</code>: TODO</li>
<li><code class="language-plaintext highlighter-rouge">objects</code>:记录文件的内容,每个文件夹名称是该object的sha1值的前两位,文件夹下的文件名称是sha1值的后18位;(tips:sha1算法,是一种加密算法,会计算当前内容的哈希值,作为object的文件名,得到的哈希值是一个用十六进制数字组成的字符串(长度为40))</li>
<li><code class="language-plaintext highlighter-rouge">refs</code>
<ul>
<li><code class="language-plaintext highlighter-rouge">heads</code>: <code class="language-plaintext highlighter-rouge">heads</code>里的就是各个分支的<code class="language-plaintext highlighter-rouge">HEAD</code>分别指向哪个<code class="language-plaintext highlighter-rouge">commit id</code>;简单说,就是 各个branch分别最新的commit是什么,这样子<code class="language-plaintext highlighter-rouge">git checkout branch</code>就可以切换到对的地方</li>
<li><code class="language-plaintext highlighter-rouge">tags</code>: 同理,这个文件夹里存的都是各个tag</li>
</ul>
</li>
</ul>
<p>那么,新建一个branch的时候,只要在<code class="language-plaintext highlighter-rouge">refs/heads</code>文件夹里新建branch 名字的文件,并将当前commit id存进去即可;</p>
<p>新建一个commit时,只要根据<code class="language-plaintext highlighter-rouge">HEAD</code>文件,找到当前的<code class="language-plaintext highlighter-rouge">branch或者tag</code> 是什么,修改里面的内容即可。</p>
<p>有点不好懂?咱给出一个git的实例,默认在一个文件夹执行<code class="language-plaintext highlighter-rouge">git init</code>后,添加一个文件并<code class="language-plaintext highlighter-rouge">commit</code>的信息, commit id为<code class="language-plaintext highlighter-rouge">017aa3d7851e8bbff78a697566b5f827b183483c</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat</span> .git/HEAD
ref: refs/heads/master
<span class="nv">$ </span><span class="nb">cat</span> .git/refs/heads/master
017aa3d7851e8bbff78a697566b5f827b183483c
</code></pre></div></div>
<p>如上,<code class="language-plaintext highlighter-rouge">HEAD</code>指向了master,而<code class="language-plaintext highlighter-rouge">master</code>的commit id正是刚刚commit的id。</p>
<h2 id="存储读取解决了那么commit怎么组织呢">存储读取解决了,那么commit怎么组织呢?</h2>
<blockquote>
<p>将当前的<code class="language-plaintext highlighter-rouge">repository</code>状态存储起来,作为commit。你可以通过<code class="language-plaintext highlighter-rouge">commit</code>恢复到任意状态,<code class="language-plaintext highlighter-rouge">git tag</code>本质也只是给这个<code class="language-plaintext highlighter-rouge">commit</code>一个<code class="language-plaintext highlighter-rouge">tag</code>(别名),<code class="language-plaintext highlighter-rouge">git branch</code> 也是一样。</p>
<p>恢复到某一个<code class="language-plaintext highlighter-rouge">commit</code>,就是将它所代表的<code class="language-plaintext highlighter-rouge">repository</code>状态恢复起来,就是将文件全部内容以及当前commit恢复到那个状态。</p>
</blockquote>
<p>上面说了,管理文件夹(repository)状态,但是文件夹是可以嵌套的,与文件不一样,需要有这层级关系,同时也要存文件内容,怎么做来区分呢?</p>
<p>我们可以引入以下概念:</p>
<ul>
<li>
<p>Tree:代表文件夹,因为<code class="language-plaintext highlighter-rouge">git init</code>时,就是把当前文件夹<code class="language-plaintext highlighter-rouge">./</code>作为项目来管理,那么接下来所有要追踪的项目无非就是<code class="language-plaintext highlighter-rouge">./</code>里的文件或者文件夹而已;</p>
</li>
<li>
<p>Blob:文件,Tree里可以包含它;</p>
</li>
</ul>
<p>关系如下图:</p>
<p><img src="https://images.xiaozhuanlan.com/photo/2020/3a033be91cacb4509dbe411db3dfb8bb.png" alt="" /></p>
<p>给点我们写的数据结构代码你看看,要注意的是,<code class="language-plaintext highlighter-rouge">tree</code>可以拥有<code class="language-plaintext highlighter-rouge">blob</code>或者<code class="language-plaintext highlighter-rouge">tree</code>,所以用了<code class="language-plaintext highlighter-rouge">union</code>;<code class="language-plaintext highlighter-rouge">parent</code>与<code class="language-plaintext highlighter-rouge">next</code>作为链表使用,作为文件夹目录管理;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct tree_entry_list {
struct tree_entry_list *next;
union {
struct tree *tree;
struct blob *blob;
} item;
struct tree_entry_list *parent;
};
struct tree {
struct tree_entry_list *entries;
};
</code></pre></div></div>
<p>而<code class="language-plaintext highlighter-rouge">commit</code>跟树一样,也是有层级的单链表,不过只有</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct commit {
struct commit *parents;
struct tree *tree;
char *commit_id[10];
char *author;
char *committer;
char *changelog;
};
</code></pre></div></div>
<p>一图胜千言,看图吧:</p>
<p><img src="https://images.xiaozhuanlan.com/photo/2020/33b652fb39f25b13d1a1d53557c9750d.png" alt="" /></p>
<h2 id="云风的游戏资源仓库及升级发布">云风的游戏资源仓库及升级发布</h2>
<p>云风参考过git的原理做过一个游戏资源仓库管理,我下面讲一下它跟git的区别,<a href="https://blog.codingnow.com/2018/08/asset_repo.html">他的文章</a>我觉得比较绕,没有背景知识的人很难看明白。</p>
<h3 id="背景">背景</h3>
<blockquote>
<p>我们的引擎的一个重要特性就是,在 PC 上开发,在移动设备上运行调试。我们需要频繁的将资源同步到设备上</p>
<p>程序以 c/s 结构运行时,在移动设备上先建立一个空的镜像仓库,同步 PC 端的资源仓库。运行流程是这样的:</p>
<p>首先在客户端启动的时候,向服务器索取一个根索引的 hash ,在本地镜像上设定根。</p>
<p>客户端请求一个文件路径时,从根开始寻找对应的目录索引文件,逐级查找。如果本地有所需的 hash 对象,就直接使用;否则向服务器请求,直到最后获得目标文件。api 的设计上,open 一个资源路径,要么返回最终的文件,要么返回一个 hash ,表示当前还缺少这个 hash 对象;这样,可以通过网络模块请求这个对象;获得该对象后,无须理会这个对象是什么,简单写入镜像仓库,然后重新前面的过程,再次请求未完成的路径,最终就能打开所需的资源文件。</p>
</blockquote>
<p>场景是:Client <- 他的游戏服务器 ,单向同步;</p>
<p>他是这样子做的,客户端的仓库是<code class="language-plaintext highlighter-rouge">key-value</code>的文件数据库,key是文件的hash,value就是文件内容;</p>
<p>同步时,会从根到具体hash全量同步文件下载到<code class="language-plaintext highlighter-rouge">数据库</code>;</p>
<p>假如客户端使用资源时,发现缺乏这个文件,就用hash去服务器拉下来。</p>
<p>换言之,因为不需要管理本地版本,并且同步到上游,所以无需在本地记录全量的版本状态</p>
<h4 id="跟git的区别">跟Git的区别:</h4>
<p>场景是:Client <-> gitHub ,双向同步;</p>
<p>git 需要本地组织commit,切换本地有但服务器没有的版本(就是离线操作) ,同时还需要将变更同步到上游。</p>
<h2 id="最后的建议">最后的建议</h2>
<p>如果看完该文,让你跃跃欲试的话,请不要用C写,请不要用C写,请不要用C写。</p>
<p>从零开始写过几个大一点项目,每次都觉得用C写项目太难受了,这次我写<code class="language-plaintext highlighter-rouge">git commit</code>时,发现要读写文件,解析内容,我发出了内心的感叹:</p>
<blockquote>
<p>太难了,不是写这个难,是C太难用了。。</p>
<p>想到我要遍历这些文件,根据目录得到tree的hash,然后还要update这棵树,把tree跟commit还要blob反序列存到文件里,还要读出来,之后还要组织链表操作,用C写就觉得百般阻挠。。。</p>
</blockquote>
最熟悉的陌生人, 5分钟快速理解HTTP2
2020-03-13T00:00:00+00:00
https://yonghaowu.github.io//2020/03/13/http2
<p><img src="https://img-blog.csdnimg.cn/20210211125314723.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hpb2hpb2h1,size_16,color_FFFFFF,t_70#pic_center" alt="公众号" /></p>
<h2 id="最熟悉的陌生人5-分钟快速理解-http2">最熟悉的陌生人:5 分钟快速理解 HTTP2</h2>
<p>from : https://mp.weixin.qq.com/s/fb02vTE884Txx6npW2mfcQ 有些图裂了,看原文比较方便~</p>
<p>最熟悉的陌生人系列,将带你快速理解熟悉的名词如:HTTP2、HTTP3、IPV6、BBR 等。</p>
<blockquote>
<p>通读 90 年代上下的论文,你会发现,在已经基本建成的计算机科学大厦中,后辈码农只要做一些零星的修补工作就行了。</p>
<p>在计算机科学晴朗天空的远处,还有几朵令人不安的小小乌云。</p>
<p>——皓尼・郝里斯( HioHio )</p>
</blockquote>
<p><img src="assets/M79Y88z.png" alt="img" /></p>
<p>而其中一朵小小乌云,就是前辈的协议制定实现得太牢靠了,就算有着诸多不足,还是用的好好的,让后辈没什么动力去创新替换。。</p>
<h3 id="http-的不足">HTTP 的不足</h3>
<p>在阅读此章时,读者可以给自己一个思考时间,锻炼设计与思考能力—— 目前在用的 HTTP 协议,你认为有哪些不足呢? 你可以重新设计一个替代它并且尽可能兼容的协议,你会怎么做呢?</p>
<p>可尝试自己写下设计,定会受益甚多。</p>
<h5 id="tcp-连接数过多">TCP 连接数过多</h5>
<p><code class="language-plaintext highlighter-rouge">HTTP1.0</code>只允许一条 tcp 链接上处理一个 request,尽管后来的 <code class="language-plaintext highlighter-rouge">HTTP1.1</code>(现在常用的版本)允许<code class="language-plaintext highlighter-rouge">pipelining</code>, 管道,通过这个管道,浏览器的多个请求可以同时发到服务器,但是服务器的响应只能够一个接着一个的返回 (但各大浏览器有些不支持 / 默认关闭,因此这功能可以说是鸡肋)。</p>
<h5 id="http-头部过多重复">HTTP 头部过多重复</h5>
<p><code class="language-plaintext highlighter-rouge">Host</code>、<code class="language-plaintext highlighter-rouge">Accept-Encoding</code>、<code class="language-plaintext highlighter-rouge">Connection</code>、<code class="language-plaintext highlighter-rouge">origin</code>、<code class="language-plaintext highlighter-rouge">content-type</code>等等一堆头部,都在不同的请求中重复出现。</p>
<p>除了浪费大量流量,还会导致 <code class="language-plaintext highlighter-rouge">TCP</code> 的初始拥塞窗口(<code class="language-plaintext highlighter-rouge">initcwnd</code>)快速满了,当多个请求准备在同一个 tcp 连接上发送时,会导致大量延迟——当<code class="language-plaintext highlighter-rouge">initcwnd >= ssthresh ( slow start threshold )</code> 时,tcp 就会进入 “拥塞避免算法”,把发送的速度调慢,避免增长过快导致网络拥塞,慢慢的增加调整到网络的最佳值。</p>
<p>当然初始拥塞窗口(initcwnd)也不能调太大来避免。</p>
<blockquote>
<p>If the initcwnd values is large, then there will be fewer RTTs required to download the same file. But we cannot set initcwnd to a huge value as the network environment and the routers also has the limitation of having limited buffers. If exceedingly large values are set, it may lead to router buffer overflows, packet loss, packet re-transmissions. So, we need to set an optimal value for the initcwnd which is directly proportional to the network bandwidth.</p>
</blockquote>
<h5 id="使用文本协议">使用文本协议</h5>
<p>文本协议尽管带来了可读性以及方便程序员 debug,但这是高性能网络程序要竭力避免的——君不见每个公司内部都要搞一个自己的二进制协议吗?二进制,每个在网络上交流的 bit 的意义都被发挥得淋漓尽致。</p>
<p>而说到 可读与 debug 的问题,自然浏览器(客户端),服务器(框架)可以帮你解决,套上一层中间层就好。</p>
<h3 id="http2-概览">HTTP2 概览</h3>
<p><code class="language-plaintext highlighter-rouge">HTTP2</code>, 为解决以上问题而生。</p>
<ul>
<li>允许多个 request/response 在同一个 tcp 链接上发送</li>
<li>高效压缩头部( http header )</li>
<li>二进制协议,真正的多路复用</li>
<li>还有自己的流量控制,保证各个 stream 不被互相干扰;</li>
<li>支持请求分优先级发送,优先级越高如核心 css、html,优先发给客户端</li>
<li>支持服务器预测并推送客户端可能需要的资源,让客户端先做缓存( server push ),榨干服务器</li>
<li>兼容 HTTP1.1 的语义,尽可能一致。</li>
</ul>
<h4 id="兼容-http11">兼容 HTTP1.1</h4>
<p>其实平常我们在用的网站都支持 HTTP2 了,如</p>
<ul>
<li>国外的 <code class="language-plaintext highlighter-rouge">google.com</code>、<code class="language-plaintext highlighter-rouge">tour.golang.org</code>、<code class="language-plaintext highlighter-rouge">facebook.com</code>、<code class="language-plaintext highlighter-rouge">stackoverflow.com</code> 、<code class="language-plaintext highlighter-rouge">shopee.sg</code></li>
<li>国内的<code class="language-plaintext highlighter-rouge">zhihu.com</code>(点赞)、<code class="language-plaintext highlighter-rouge">v2ex.com</code>、<code class="language-plaintext highlighter-rouge">vgtime.com</code>(我擦,这个做的 UI 看起来有点渣的游戏资讯网站竟然支持你敢信,人家 gcores 都不支持)、<code class="language-plaintext highlighter-rouge">youku.com</code></li>
</ul>
<p>而想找一些不支持的,找一些小型网站就好,如 <code class="language-plaintext highlighter-rouge">yonghaowu.github.io</code>、<code class="language-plaintext highlighter-rouge">gcores.com</code>,<code class="language-plaintext highlighter-rouge">douban.com</code>、<code class="language-plaintext highlighter-rouge">bilibili.com/</code>,还有臭名昭著的 <code class="language-plaintext highlighter-rouge">baidu.com</code></p>
<p>当然,这里说不支持时,只是说这个域名不支持,他可能 api 是用另外的域名然后是支持的。</p>
<h6 id="升级-http2">升级 HTTP2</h6>
<p>兼容,或者说客户端要求升级到 HTTP2,主要有两种方法:</p>
<ul>
<li>
<p>客户端的</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP header
</code></pre></div> </div>
<p>的</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Upgrade
</code></pre></div> </div>
<p>指定</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>h2c
</code></pre></div> </div>
<p>(</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP/2 ClearText
</code></pre></div> </div>
<p>)</p>
<ul>
<li>如你所知,<code class="language-plaintext highlighter-rouge">Connection: Upgrade</code>与<code class="language-plaintext highlighter-rouge">Upgrade: websocket</code>,Websocket 就是这样子变换协议的;</li>
</ul>
</li>
<li>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ALPN ( Application Layer Protocol Negotiation,应用层协议协商)
</code></pre></div> </div>
<p>,TLS 的扩展功能</p>
<ul>
<li>客户端在建立 TLS 连接的 Client Hello 握手中,通过 ALPN 扩展列出了自己支持的各种应用层协议</li>
<li>如果服务端支持 HTTP/2,在 Server Hello 中指定 ALPN 的结果为 <code class="language-plaintext highlighter-rouge">h2</code> 就可以了</li>
<li>如果服务端不支持 HTTP/2,从客户端的 ALPN 列表中选一个自己支持的即可</li>
</ul>
</li>
</ul>
<p>而一般你看现在的网站请求,都用第二种方式了,因为第一种方式服务端接收到后还需要返回<code class="language-plaintext highlighter-rouge">101 状态码 Switching Protocols</code>告知客户端,客户端再发送 http2 的数据。</p>
<h3 id="http2-的-帧-frame-">HTTP2 的 帧( frame )</h3>
<p>HTTP2 中二进制协议的基本单元叫 frame (帧),不同 frame 有不同作用,如:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">SETTING</code>帧:建立连接时,向对方传达一些配置信息如是否开启 server push 功能、最大帧 size 等等(牢记,下文不累述此);</li>
<li><code class="language-plaintext highlighter-rouge">HEADERS</code>帧:发送 http 的 request 或者 response 的头部;</li>
<li><code class="language-plaintext highlighter-rouge">CONTINUATION</code>帧:headers 要跨越多个帧,用此来指示头部上一个<code class="language-plaintext highlighter-rouge">HEADERS</code>;本质就是<code class="language-plaintext highlighter-rouge">HEADERS</code>帧,但是为了轻松处理,就用明确的类型来区分这种情况;</li>
<li><code class="language-plaintext highlighter-rouge">DATA</code>帧:发送 body 数据用;</li>
<li><code class="language-plaintext highlighter-rouge">PUSH_PROMISE</code> 帧:用来告知对端初始化哪些数据,就是以上说到的 <code class="language-plaintext highlighter-rouge">server push</code> 功能</li>
<li><code class="language-plaintext highlighter-rouge">WINDOW_UPDATE</code>帧:用来做流量控制</li>
</ul>
<p>等。</p>
<p>帧的格式如下,熟悉二进制协议的你对此想必很清晰:</p>
<ul>
<li>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+-----------------------------------------------+
| Length (24) |
+---------------+---------------+---------------+
| Type (8) | Flags (8) |
+-+-------------+---------------+-------------------------------+
|R| Stream Identifier (31) |
+=+=============================================================+
| Frame Payload (0...) ...
+---------------------------------------------------------------+
</code></pre></div> </div>
<ul>
<li><code class="language-plaintext highlighter-rouge">length</code>: <code class="language-plaintext highlighter-rouge">frame payload</code> 的长度;</li>
<li><code class="language-plaintext highlighter-rouge">type</code>:<code class="language-plaintext highlighter-rouge">frame</code> 的类型;</li>
<li><code class="language-plaintext highlighter-rouge">flag</code>: 保留给<code class="language-plaintext highlighter-rouge">frame</code> 的类型使用;</li>
<li><code class="language-plaintext highlighter-rouge">R</code>: 保留的一个 bit,没有任何作用;</li>
<li><code class="language-plaintext highlighter-rouge">Stream Identifier</code>:unsigned 31 位整数<code class="language-plaintext highlighter-rouge">id</code>,用来区分 stream ;</li>
<li><code class="language-plaintext highlighter-rouge">Frame Payload</code>: frame 携带的可变长数据,可为空;</li>
</ul>
<p>以上 6 种东西,<code class="language-plaintext highlighter-rouge">Frame Payload</code> 可以没有,但是其他必须有。</p>
<p>所以所有 frame 必定会有至少 <code class="language-plaintext highlighter-rouge">24 + 8 + 8 + 1 + 31 + (0…) = 72 位</code>的数据。</p>
<p>一个经典的 http 请求在 http2 中对应如下,可以看到 <code class="language-plaintext highlighter-rouge">HEADERS</code>跟 <code class="language-plaintext highlighter-rouge">DATA</code> 两个 frame:</p>
<p><img src="https://www.v2ex.com/t/assets/ae09920e853bee0b21be83f8e770ba01.svg" alt="Figure 12-1. HTTP/2 binary framing layer" /></p>
<p>值得注意的是,当 data 过大的时候,http2 的 rfc 没有规定 data frame 应该拆分与否(翻了一大堆资料都没有找到)。</p>
<p>然而去用一些工具如 nghttp 去看详细过程,可看到 data frame 都是拆开一个个的,原因就是为了多路复用。这</p>
<p><code class="language-plaintext highlighter-rouge">$ nghttp -v -n --no-dep -w 14 -a https://www.vgtime.com</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span> 0.063] recv <span class="o">(</span><span class="nv">stream_id</span><span class="o">=</span>9<span class="o">)</span> eagleid: 2ff6019a15691588216324974e
<span class="o">[</span> 0.063] recv <span class="o">(</span><span class="nv">stream_id</span><span class="o">=</span>9<span class="o">)</span> content-encoding: <span class="nb">gzip</span>
<span class="o">[</span> 0.063] recv HEADERS frame <<span class="nv">length</span><span class="o">=</span>188, <span class="nv">flags</span><span class="o">=</span>0x04, <span class="nv">stream_id</span><span class="o">=</span>9>
<span class="p">;</span> END_HEADERS
<span class="o">(</span><span class="nv">padlen</span><span class="o">=</span>0<span class="o">)</span>
<span class="p">;</span> First response header
<span class="o">[</span> 0.063] recv DATA frame <<span class="nv">length</span><span class="o">=</span>8192, <span class="nv">flags</span><span class="o">=</span>0x00, <span class="nv">stream_id</span><span class="o">=</span>9>
<span class="o">[</span> 0.063] recv DATA frame <<span class="nv">length</span><span class="o">=</span>464, <span class="nv">flags</span><span class="o">=</span>0x00, <span class="nv">stream_id</span><span class="o">=</span>9>
<span class="o">[</span> 0.063] recv DATA frame <<span class="nv">length</span><span class="o">=</span>2510, <span class="nv">flags</span><span class="o">=</span>0x00, <span class="nv">stream_id</span><span class="o">=</span>9>
<span class="o">[</span> 0.063] recv DATA frame <<span class="nv">length</span><span class="o">=</span>10, <span class="nv">flags</span><span class="o">=</span>0x01, <span class="nv">stream_id</span><span class="o">=</span>9>
<span class="p">;</span> END_STREAM
</code></pre></div> </div>
<p>所以一个大的请求如下图,常见的帧就是每一个 <code class="language-plaintext highlighter-rouge">Frame Header</code> 接一个 <code class="language-plaintext highlighter-rouge">Frame Body</code>。</p>
<p><img src="assets/Binary_framing2-4853753.png" alt="img" /></p>
<p>帧的大小范围规定为 <code class="language-plaintext highlighter-rouge">2 的 14 次方 (16,384)</code> 到 <code class="language-plaintext highlighter-rouge">2 的 24 次方-1 (16,777,215)</code> 字节,也就是大概 <code class="language-plaintext highlighter-rouge">16KB 到 16MB</code>。</p>
<p>但若双方没有协议,一般默认为 16Kb,假如<code class="language-plaintext highlighter-rouge">HEADERS</code>帧不够装完头部时,就用第二个 <code class="language-plaintext highlighter-rouge">CONTINUATION</code>帧来装,</p>
<p>所以你看到可以有多个 <code class="language-plaintext highlighter-rouge">CONTINUATETION</code>帧下有省略号,因为可以有多个。</p>
<h3 id="流-stream-">流( stream )</h3>
<p>流在 HTTP2 一条连接中,在客户端与服务端之间,双向交换帧( frame )。</p>
<p>简单说,客户端与服务端之间相互发送的帧,都通过一个个独立流来传输,多个流可以在同一 http2 连接中并发,而每个流都有一个 ID ( Stream Identifier ),frame 就是通过此来识别流。</p>
<p>流你可以理解为一个抽象概念,就是为了区分不同的请求,用于多路复用。</p>
<p>流的状态机如下:</p>
<p><img src="assets/iUPIXLC.png" alt="img" /></p>
<p>我们常见的 HTTP 请求就是走黄色的线:</p>
<p><code class="language-plaintext highlighter-rouge">idle</code>状态 -> 发送 <code class="language-plaintext highlighter-rouge">HEADER</code>帧后变成<code class="language-plaintext highlighter-rouge">OPEN</code> -> 发送完数据后发送 <code class="language-plaintext highlighter-rouge">END_STREAM</code>代表发完 -> 变成 <code class="language-plaintext highlighter-rouge">half closed</code>状态 -> 等待对方发送 <code class="language-plaintext highlighter-rouge">END_STREAM</code>代表对方发完 。</p>
<p>你会发现这个流程非常像 <code class="language-plaintext highlighter-rouge">TCP 的四次挥手</code>,因为本质都是自己关闭流后,要等待对方关闭并自己来确认。</p>
<p>当然,也会有像四次挥手一样的<code class="language-plaintext highlighter-rouge">RESET</code> 一样 <code class="language-plaintext highlighter-rouge">reset stream 的功能</code>,我就不累述了。</p>
<h4 id="stream-流量控制">Stream 流量控制</h4>
<p>HTTP2 的 Stream 有流量控制功能,HTTP2 的接收方通过 <a href="https://http2.github.io/http2-spec/#WINDOW_UPDATE">WINDOW_UPDATE</a> 帧告诉对方自己准备接收多少字节的数据,注意只有 <code class="language-plaintext highlighter-rouge">DATA 帧</code>才会受限制,因为其他帧都不大,而且也比较重要。</p>
<h4 id="stream-优先级">Stream 优先级</h4>
<p>客户端可以在开启一个流时,通过设置在<code class="language-plaintext highlighter-rouge">HEADER 帧</code>里的<code class="language-plaintext highlighter-rouge">PRIORITY</code>这个 flag,来指定流的优先级。这样子就可以做到优先级越高如核心 css、html,优先发给客户端</p>
<h3 id="server-push">Server Push</h3>
<p><code class="language-plaintext highlighter-rouge">HTTP2</code>打破了以往 <code class="language-plaintext highlighter-rouge">HTTP1</code> 一问一答的范式,允许服务器主动往客户端推数据了,但值得注意的是,这依然不能代替 <code class="language-plaintext highlighter-rouge">Websoket</code>,两者是不等价的,除非你自己重新实现 http2 客户端服务端的功能——也就是改 HTTP2 协议了。</p>
<p>服务器可以通过 <code class="language-plaintext highlighter-rouge">PUSH_PROMISE</code>帧,把预估客户端可能需要的资源,在其没有请求前直接发送给对方,让对方缓存。如下图就直接发了 <code class="language-plaintext highlighter-rouge">styles.css</code>给对方。</p>
<p><img src="https://www.v2ex.com/t/assets/server-push-response.png" alt="Web Server Communication with HTTP/2 server push." /></p>
<h3 id="头部压缩-hpack-">头部压缩( HPACK )</h3>
<p><code class="language-plaintext highlighter-rouge">HPACK</code>就是专门用来处理重复冗余的头部的,对这个优化,自然就想到查表法——客户端发送请求前,在内部创建一个哈希表,索引对应着头部与值,并将此对应表发送供给服务器;服务器首次接收到后,也维护一个一模一样的表,之后有重复头部时,客户端直接发索引值即可。</p>
<h2 id="后记">后记</h2>
<p>拖拖拉拉,写了一两周总算把这篇学习笔记写完了,相比网上很多文章或者书籍(比如网上很多人没讲明白流是什么,frame 如何分段等),我觉得这篇笔记是系统性的且非常符合不熟悉 HTTP2 的同学理解它是什么的。</p>
<p>有很多知识是精简了的,以后看读者反馈再补充。</p>
<p>如果您觉得写的好,对您有用,不妨用行动(你懂的)多多支持~</p>
</li>
</ul>