发表于
字数统计: | 阅读时长 ≈

发表于 | 分类于
字数统计: | 阅读时长 ≈

pprof

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
heap profile: 13: 764144560 [139616: 4600982112] @ heap/1048576
1: 258760704 [1: 258760704] @ 0x947570 0x946625 0x94e9bc 0x46abf1
#	0x94756f	main.(*MySQLDecoder).decodeClientPacket+0xf2f	/home/q/gopath/src/my-sniffer/decode.go:54
#	0x946624	main.(*MySQLDecoder).DecodeClientPacket+0x134	/home/q/gopath/src/my-sniffer/decode.go:38
#	0x94e9bb	main.(*Sniffer).parse+0x37b			/home/q/gopath/src/my-sniffer/sniffer.go:88

1: 207003648 [2: 414007296] @ 0x947570 0x946625 0x94e9bc 0x46abf1
#	0x94756f	main.(*MySQLDecoder).decodeClientPacket+0xf2f	/home/q/gopath/src/my-sniffer/decode.go:54
#	0x946624	main.(*MySQLDecoder).DecodeClientPacket+0x134	/home/q/gopath/src/my-sniffer/decode.go:38
#	0x94e9bb	main.(*Sniffer).parse+0x37b			/home/q/gopath/src/my-sniffer/sniffer.go:88

1: 165601280 [3: 496803840] @ 0x947570 0x946625 0x94e9bc 0x46abf1
#	0x94756f	main.(*MySQLDecoder).decodeClientPacket+0xf2f	/home/q/gopath/src/my-sniffer/decode.go:54
#	0x946624	main.(*MySQLDecoder).DecodeClientPacket+0x134	/home/q/gopath/src/my-sniffer/decode.go:38
#	0x94e9bb	main.(*Sniffer).parse+0x37b			/home/q/gopath/src/my-sniffer/sniffer.go:88

1: 54255616 [1: 54255616] @ 0x948a55 0x9464cf 0x94eb24 0x46abf1
#	0x948a54	main.(*MySQLDecoder).decodeServerPacket+0xba4	/home/q/gopath/src/my-sniffer/decode.go:218
#	0x9464ce	main.(*MySQLDecoder).DecodeServerPacket+0x13e	/home/q/gopath/src/my-sniffer/decode.go:27
#	0x94eb23	main.(*Sniffer).parse+0x4e3			/home/q/gopath/src/my-sniffer/sniffer.go:83

1: 43401216 [2: 86802432] @ 0x948a55 0x9464cf 0x94eb24 0x46abf1
#	0x948a54	main.(*MySQLDecoder).decodeServerPacket+0xba4	/home/q/gopath/src/my-sniffer/decode.go:218
#	0x9464ce	main.(*MySQLDecoder).DecodeServerPacket+0x13e	/home/q/gopath/src/my-sniffer/decode.go:27
#	0x94eb23	main.(*Sniffer).parse+0x4e3			/home/q/gopath/src/my-sniffer/sniffer.go:83

1: 34717696 [3: 104153088] @ 0x948a55 0x9464cf 0x94eb24 0x46abf1
#	0x948a54	main.(*MySQLDecoder).decodeServerPacket+0xba4	/home/q/gopath/src/my-sniffer/decode.go:218
#	0x9464ce	main.(*MySQLDecoder).DecodeServerPacket+0x13e	/home/q/gopath/src/my-sniffer/decode.go:27
#	0x94eb23	main.(*Sniffer).parse+0x4e3			/home/q/gopath/src/my-sniffer/sniffer.go:83

1: 196608 [1: 196608] @ 0x4f92eb 0x4fb0b2 0x9470ef 0x9470a3 0x947069 0x946625 0x94e9bc 0x46abf1
#	0x4f92ea	log.(*Logger).Output+0x38a					/usr/lib/golang/src/log/log.go:168
#	0x4fb0b1	my-sniffer/vendor/github.com/ngaut/log.(*logger).logf+0x1e1	/home/q/gopath/src/my-sniffer/vendor/github.com/ngaut/log/log.go:291
#	0x9470ee	my-sniffer/vendor/github.com/ngaut/log.(*logger).Infof+0xaae	/home/q/gopath/src/my-sniffer/vendor/github.com/ngaut/log/log.go:333
#	0x9470a2	my-sniffer/vendor/github.com/ngaut/log.Infof+0xa62		/home/q/gopath/src/my-sniffer/vendor/github.com/ngaut/log/log.go:84
#	0x947068	main.(*MySQLDecoder).decodeClientPacket+0xa28			/home/q/gopath/src/my-sniffer/decode.go:118
#	0x946624	main.(*MySQLDecoder).DecodeClientPacket+0x134			/home/q/gopath/src/my-sniffer/decode.go:38
#	0x94e9bb	main.(*Sniffer).parse+0x37b					/home/q/gopath/src/my-sniffer/sniffer.go:88

1: 163840 [1: 163840] @ 0x94fc46 0x9501dd 0x46abf1
#	0x94fc45	main.(*Sniffer).manageChannel+0x4f5	/home/q/gopath/src/my-sniffer/sniffer.go:194
#	0x9501dc	main.(*Sniffer).ServerCron+0x23c	/home/q/gopath/src/my-sniffer/sniffer.go:244

1: 18432 [1: 18432] @ 0x87e913 0x87e956 0x93c68c 0x49f316 0x49ead4 0x87dc90 0x926511 0x93b533 0x938fad 0x46abf1
#	0x87e912	my-sniffer/vendor/github.com/rcrowley/go-metrics.newExpDecaySampleHeap+0x72			/home/q/gopath/src/my-sniffer/vendor/github.com/rcrowley/go-metrics/sample.go:544
#	0x87e955	my-sniffer/vendor/github.com/rcrowley/go-metrics.NewExpDecaySample+0xb5				/home/q/gopath/src/my-sniffer/vendor/github.com/rcrowley/go-metrics/sample.go:56
#	0x93c68b	my-sniffer/vendor/github.com/Shopify/sarama.getOrRegisterHistogram.func1+0x3b			/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/metrics.go:21
#	0x49f315	reflect.Value.call+0x5f5									/usr/lib/golang/src/reflect/value.go:460
#	0x49ead3	reflect.Value.Call+0xb3										/usr/lib/golang/src/reflect/value.go:321
#	0x87dc8f	my-sniffer/vendor/github.com/rcrowley/go-metrics.(*StandardRegistry).GetOrRegister+0x28f	/home/q/gopath/src/my-sniffer/vendor/github.com/rcrowley/go-metrics/registry.go:99
#	0x926510	my-sniffer/vendor/github.com/Shopify/sarama.getOrRegisterHistogram+0x60				/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/metrics.go:20
#	0x93b532	my-sniffer/vendor/github.com/Shopify/sarama.(*Broker).Open.func1+0x7a2				/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/broker.go:183
#	0x938fac	my-sniffer/vendor/github.com/Shopify/sarama.withRecover+0x4c					/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/utils.go:45

1: 16384 [8053: 131940352] @ 0x86f83a 0x7edb28 0x7edda3 0x46abf1
#	0x86f839	my-sniffer/vendor/github.com/google/gopacket/pcap.(*Handle).ReadPacketData+0x189	/home/q/gopath/src/my-sniffer/vendor/github.com/google/gopacket/pcap/pcap.go:247
#	0x7edb27	my-sniffer/vendor/github.com/google/gopacket.(*PacketSource).NextPacket+0x47		/home/q/gopath/src/my-sniffer/vendor/github.com/google/gopacket/packet.go:801
#	0x7edda2	my-sniffer/vendor/github.com/google/gopacket.(*PacketSource).packetsToChannel+0x82	/home/q/gopath/src/my-sniffer/vendor/github.com/google/gopacket/packet.go:818

1: 8192 [67: 548864] @ 0x946ccf 0x946625 0x94e9bc 0x46abf1
#	0x946cce	main.(*MySQLDecoder).decodeClientPacket+0x68e	/home/q/gopath/src/my-sniffer/decode.go:101
#	0x946624	main.(*MySQLDecoder).DecodeClientPacket+0x134	/home/q/gopath/src/my-sniffer/decode.go:38
#	0x94e9bb	main.(*Sniffer).parse+0x37b			/home/q/gopath/src/my-sniffer/sniffer.go:88

1: 896 [1: 896] @ 0x92582b 0x9184b5 0x8ff6a4 0x8fd520 0x90a374 0x9078c0 0x905987 0x8f630d 0x952977 0x952b28 0x46abf1
#	0x92582a	my-sniffer/vendor/github.com/Shopify/sarama.(*MetadataResponse).decode+0xda	/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/metadata_response.go:169
#	0x9184b4	my-sniffer/vendor/github.com/Shopify/sarama.versionedDecode+0xb4		/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/encoder_decoder.go:79
#	0x8ff6a3	my-sniffer/vendor/github.com/Shopify/sarama.(*Broker).sendAndReceive+0x203	/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/broker.go:719
#	0x8fd51f	my-sniffer/vendor/github.com/Shopify/sarama.(*Broker).GetMetadata+0x6f		/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/broker.go:287
#	0x90a373	my-sniffer/vendor/github.com/Shopify/sarama.(*client).tryRefreshMetadata+0x4c3	/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/client.go:784
#	0x9078bf	my-sniffer/vendor/github.com/Shopify/sarama.(*client).RefreshMetadata+0xaf	/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/client.go:442
#	0x905986	my-sniffer/vendor/github.com/Shopify/sarama.NewClient+0x616			/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/client.go:159
#	0x8f630c	my-sniffer/vendor/github.com/Shopify/sarama.NewAsyncProducer+0x4c		/home/q/gopath/src/my-sniffer/vendor/github.com/Shopify/sarama/async_producer.go:111
#	0x952976	main.(*KafkaWriter).Producer+0xa6						/home/q/gopath/src/my-sniffer/write.go:144
#	0x952b27	main.(*KafkaWriter).Write+0xc7							/home/q/gopath/src/my-sniffer/write.go:175

1: 48 [12729: 610992] @ 0x41996d 0x7e5b9c 0x94c484 0x94e92a 0x46abf1
#	0x7e5b9b	my-sniffer/vendor/github.com/streamrail/concurrent-map.New+0x7b	/home/q/gopath/src/my-sniffer/vendor/github.com/streamrail/concurrent-map/concurrent_map.go:24

764144560-258760704-207003648-165601280-54255616-43401216-34717696-196608-163840-18432-16384-8192-896-48=0

http://xxxx:8888/debug/pprof/heap?debug=1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# runtime.MemStats
# Alloc = 6911928
# TotalAlloc = 19160914800
# Sys = 72373071
# Lookups = 0
# Mallocs = 201435665
# Frees = 201407065
# HeapAlloc = 6911928
# HeapSys = 65994752
# HeapIdle = 56852480
# HeapInuse = 9142272
# HeapReleased = 52936704
# HeapObjects = 28600
# Stack = 1114112 / 1114112
# MSpan = 106488 / 196608
# MCache = 6944 / 65536
# BuckHashSys = 1616868
# GCSys = 2394112
# OtherSys = 991083
# NextGC = 11362960
# LastGC = 1574322082513879647
# PauseNs = [189598 49953 92502 32422 38027 82085 100885 229316 76361 72260 78828 39619 149152 46332 70705 17145 16763 58004 64772 28182 78964 723741 61377 82393 95501 14008 20076 61063 43738 26423 80042 38541 71009 56981 28659 28020 71551 31313 98150 73792 79719 23929 99861 22701 120759 22477 68964 42352 11666 27739 59629 43175 31219 30039 81174 27849 25645 151912 49662 10258 22584 75402 63296 13006 43117 178897 31601 140531 28393 122639 85584 101906 10499 61339 126044 62926 16444 23959 49294 58229 18315 22263 42363 11487 13774 64256 78467 55070 17849 123218 49329 41235 15583 27532 22365 138046 143495 82689 83776 22407 16361 72739 30012 88179 53863 57042 19235 36012 49999 10093 150366 57210 74770 33872 17570 80342 33627 130745 45451 44563 68349 91948 80106 96768 123250 68130 37997 85617 52326 105215 79923 59876 82770 34976 71308 75440 74769 52477 81582 28035 26408 81097 51790 87575 83188 97461 74315 14031 63507 60049 70288 26258 30603 70501 25757 76272 54131 11315 15774 47247 28143 16491 44517 397021 109310 75557 38932 54458 57195 101318 13764 12970 61384 70465 74954 34697 74424 49610 546274 37799 21334 23761 26868 25757 67495 37693 48941 31569 11449 49646 26411 28535 17298 60990 14267 45562 81123 95301 51616 41552 68802 69589 80352 30789 85043 150628 9825 48716 31399 17660 90074 29268 58238 66220 21357 36999 83608 49139 10383 68450 52035 18896 22419 82571 20467 20933 49959 50114 24937 62938 17763 70881 29038 88511 33083 12115 39858 55640 25758 85965 25077 163754 28138 14249 57188 80709 53402 62238 78211 51991 12517 70669 66887 94401 65900 22837]
# PauseEnd = [1574322080390712046 1574322080402601456 1574322080417597815 1574322080430075401 1574322080448743679 1574322080481843308 1574322080503285042 1574322080521567097 1574322080537619917 1574322080559570756 1574322080588332615 1574322080605197934 1574322080618694803 1574322080636061669 1574322080657184934 1574322080684348212 1574322080705329009 1574322080719469909 1574322080736620249 1574322080755391983 1574322080777142378 1574322080801636089 1574322080825558326 1574322080836895171 1574322080849027987 1574322080865581420 1574322080878967168 1574322080900290388 1574322080919367030 1574322080940890269 1574322080974232859 1574322080988857538 1574322081002500813 1574322081017118019 1574322081035119959 1574322081052533660 1574322081065488769 1574322081083060618 1574322081103290742 1574322081117706184 1574322081132701465 1574322081153895098 1574322081177445475 1574322081196174521 1574322081209746571 1574322081223541458 1574322081238549400 1574322081253663527 1574322081279011933 1574322081295221475 1574322081312807380 1574322081329454242 1574322081349673665 1574322081382931596 1574322081398104595 1574322081411675607 1574322081423663058 1574322081435591228 1574322081452690208 1574322081482717475 1574322081506495585 1574322081523191972 1574322081539546871 1574322081560526634 1574322081585884075 1574322081604924504 1574322081620393778 1574322081636910237 1574322081649709918 1574322081671440093 1574322081692351239 1574322081714954880 1574322081736383517 1574322081759477966 1574322081780396845 1574322081809275402 1574322081832646903 1574322081844973974 1574322081859919543 1574322081878146929 1574322081894049877 1574322081918470164 1574322081942839690 1574322081962122724 1574322081973635354 1574322081991546597 1574322082006185907 1574322082020228404 1574322082043356433 1574322082062069998 1574322082092364700 1574322082111501695 1574322082127799248 1574322082141316859 1574322082157849501 1574322082177158998 1574322082190978605 1574322082206080529 1574322082220254360 1574322082238405644 1574322082255306291 1574322082287696098 1574322082300844344 1574322082315626425 1574322082327934125 1574322082342279708 1574322082359464097 1574322082403441800 1574322082421731674 1574322082447831735 1574322082472752623 1574322082486526178 1574322082502363044 1574322082513879647 1574322077489750221 1574322077502583900 1574322077515159633 1574322077531276251 1574322077553955480 1574322077574258541 1574322077596114790 1574322077613266937 1574322077625509198 1574322077640497140 1574322077674765140 1574322077702541016 1574322077721769598 1574322077738605211 1574322077775075178 1574322077799454347 1574322077822280173 1574322077835197697 1574322077850735655 1574322077882828316 1574322077916692835 1574322077938708711 1574322077973382082 1574322077990437988 1574322078006060345 1574322078018689112 1574322078030868740 1574322078048097314 1574322078077412649 1574322078099010773 1574322078116051403 1574322078133467101 1574322078145929683 1574322078164972882 1574322078193059776 1574322078218462494 1574322078243964527 1574322078271299425 1574322078296295965 1574322078311903980 1574322078323289651 1574322078340564935 1574322078355041152 1574322078379113651 1574322078397064652 1574322078410283359 1574322078425451781 1574322078461478242 1574322078489565281 1574322078509050699 1574322078520379771 1574322078534615753 1574322078551371364 1574322078569855252 1574322078590158746 1574322078610728096 1574322078632052560 1574322078653213142 1574322078689313302 1574322078706298386 1574322078719347046 1574322078733274403 1574322078757892267 1574322078807458635 1574322078829797069 1574322078847526699 1574322078863212705 1574322078893213248 1574322078912050577 1574322078932015146 1574322078952862198 1574322078968937939 1574322078983422259 1574322079003764382 1574322079024587932 1574322079077176784 1574322079096946991 1574322079115658208 1574322079129207783 1574322079143182694 1574322079157474200 1574322079193690884 1574322079206345901 1574322079226361253 1574322079242177073 1574322079260978719 1574322079314650080 1574322079329493542 1574322079345056294 1574322079368803814 1574322079387376484 1574322079408767141 1574322079428095837 1574322079449106465 1574322079467998999 1574322079485391468 1574322079499939713 1574322079512307848 1574322079524423908 1574322079544692641 1574322079572892982 1574322079601073467 1574322079616249315 1574322079633470438 1574322079647391778 1574322079672249349 1574322079693840219 1574322079710479969 1574322079723083791 1574322079740765000 1574322079761293570 1574322079789464150 1574322079807361784 1574322079829103390 1574322079849831253 1574322079871286517 1574322079892161883 1574322079912346685 1574322079929479039 1574322079944339751 1574322079983735126 1574322079996146439 1574322080012304729 1574322080025186914 1574322080042884286 1574322080075591239 1574322080092800684 1574322080109987218 1574322080130215766 1574322080148307817 1574322080166223553 1574322080184252577 1574322080201494803 1574322080216216121 1574322080238095235 1574322080254460660 1574322080274530708 1574322080292243817 1574322080311767013 1574322080326532899 1574322080345139037 1574322080374701044]
# NumGC = 7794
# NumForcedGC = 0
# GCCPUFraction = 0.10706624962106928
# DebugGC = false

结果分析:

sys

Sys 是下面列出的 XSys 字段的综合。Sys 维护着为 Go 运行时预留的虚拟内存空间地址，里面包含了：堆、栈，以及其他内部数据结构。

1
2
3
4
5
Sys(72373071) = HeapSys(65994752)+ GCSys(2394112) + OtherSys(991083) + BuckHashSys(1616868)+ StackSys(1114112) + MSpanSys(196608) + MCacheSys(65536)

Stack = 1114112(StackInuse) / 1114112(StackSys)
MSpan = 106488(MSpanInuse) / 196608(MSpanSys)
MCache = 6944(MCacheInuse) / 65536(MCacheSys)

HeapSys: 从操作系统获得的堆内存大小,虚拟内存空间为堆保留的大小，包括还没有被使用的
StackSys: 从操作系统取得的栈内存大小
MSpanSys: 从操作系统为mspan获取的内存字节数
MCacheSys: 从操作系统为mcache获取的内存字节数
BuckHashSys: 在profiling bucket hash tables中的内存字节数
GCSys: 垃圾回收元数据使用的内存字节数
OtherSys: off-heap的杂项内存字节数.

HeapSys

HeapSys：程序向应用程序申请的内存
HeapAlloc：堆上目前分配的内存
HeapIdle：堆上目前没有使用的内存
HeapReleased：回收到操作系统的内存

  • HeapSys == HeapIdle + HeapInuse 代表了作为堆用途的内存空间。堆内存以Span为单元进行管理，从Span中分配内存给对象，分为Idle（没有分配对象）和Inuse（有对象）两种类型。
  • HeapAlloc == Alloc 代表了堆上的未回收对象的实际占用空间（包括可回收部分）。注意，这和HeapInuse是有区别的。HeapInuse代表了分配对象的Span的大小，而HeapAlloc代表了实际对象的大小，因此可以认为一个是粗统计，一个是细统计，HeapAlloc <= HeapInuse
  • StackSys 代表协程栈大小，其实是Stack Span的大小，可以和Idle Span互相转换。不同版本Go语言的最小协程栈大小是不同的，这次问题里面的协程栈大小为8K，因此30万协程2.4G，和图中的Stack部分相符，但还不足以解释剩下6G的堆内存占用量。

命令行pprof命令

查看30秒钟的CPU状态信息(执行后会卡30s,等就行了)

1
go tool pprof http://localhost:6060/debug/pprof/profile

查看堆状态信息

1
go tool pprof http://localhost:6060/debug/pprof/heap

-source 命令配置源码地址[可以定位到具体的代码行]

top,list 等常用命令

https://cloud.tencent.com/developer/section/1143647

https://www.dazhuanlan.com/2019/09/29/5d907f1435d79/
https://colobu.com/2019/08/28/go-memory-leak-i-dont-think-so

Go字符串的内存布局

https://blog.csdn.net/pplin/article/details/70241075
https://draveness.me/golang/datastructure/golang-string.html

https://i6448038.github.io/2019/05/18/golang-mem/

发表于 | 分类于
字数统计: | 阅读时长 ≈

Hadoop MapReduce 源于 Google 在2004年12月份发表的 MapReduce 论文。Hadoop MapReduce 其实就是 Google MapReduce 的一个克隆版本

MapReduce 入门

MapReduce是一种编程模型,其思想来自于函数式编程，和Python，Lisp语言中的map和reduce函数类似,用于大规模数据集的分布式运算
python map函数

1
2
3
4
5
6
7
8
9
10
11
>>>def square(x) :            # 计算平方数
...     return x ** 2
... 
>>> map(square, [1,2,3,4,5])   # 计算列表各个元素的平方
[1, 4, 9, 16, 25]
>>> map(lambda x: x ** 2, [1, 2, 3, 4, 5])  # 使用 lambda 匿名函数
[1, 4, 9, 16, 25]
 
# 提供了两个列表，对相同位置的列表数据进行相加
>>> map(lambda x, y: x + y, [1, 3, 5, 7, 9], [2, 4, 6, 8, 10])
[3, 7, 11, 15, 19]

Python reduce() 函数

1
2
3
4
5
6
7
>>>def add(x, y) :            # 两数相加
...     return x + y
... 
>>> reduce(add, [1,2,3,4,5])   # 计算列表和：1+2+3+4+5
15
>>> reduce(lambda x, y: x+y, [1,2,3,4,5])  # 使用 lambda 匿名函数
15

7个实例全面掌握Hadoop MapReduce

MapReduce进阶

MapReduce工作过程

  1. MapReduce框架对输入的文件数据分成M片，每份数据的大小为16~64MB（可由用户配置）。
  2. 在多台机器上开始运行User Program：包括一个master、多个map worker和多个reduce worker。
  3. master主要负责map worker和reduce worker的状态管理和任务分发。
  4. map worker从GFS读取分配到的文件数据，并进行相应的处理。MapReduce框架的调度会尽量使map worker运行的机器与数据靠近，以提高数据传输的效率。所以，数据传输可以是本地，也可能是网络。
  5. map worker的输出缓存在内存中，并定期刷到本地磁盘上。这些中间数据的位置信息会通过心跳信息告诉master，master记下这些信息后，通知reduce worker。数据存储在本地。
  6. reduce worker通过RPC从map worker读取需要的中间数据。数据通过网络传输。
  7. reduce worker对中间数据进行“合并”处理后，输出结果。

容灾

MapReduce 如何容灾是其最重要的部分，对于故障我们可以分为 worker故障和master故障，worker故障又可以分为 map worker和reduce worker。

Worker故障

Master 通过心跳的机制来检测worker故障

  • map worker
    • map任务执行完成后宕机：因为中间数据存储在本地磁盘，需要重新执行。
    • map任务执行完成前宕机：需要重新执行
  • reduce worker
    • reduce任务执行完成后宕机：因为数据存储在GFS，不需要重新执行
    • reduce任务执行完成前宕机：需要重新执行，输出文件可以覆盖原来的（文件名一样）

Master故障

master宕机，任务失败,可以简单的通过周期写快照的方式来处理master故障

优化

  • 局部性：MapReduce用于大数据集的处理，其主要瓶颈是网络带宽。通过优化调度，可以让执行MapReduce任务的机器尽可能靠近机器。（同一机器==>同一机架==>同一机房…）
  • 任务粒度：执行MapReduce任务的过程其实就是M个Map任务+R个Reduce任务。M和R必须比机器数大很多才会有利于负载均衡
  • 备份任务：当MapReduce任务即将执行完成时，MapReduce框架会针对那些还在执行的任务，启动一个对应的备份任务。之后，只要主任务或备份任务执行完成，MapReduce任务就完成了。这样可以有效避免整个MapReduce任务被少部分比较慢的机器拖死

mapReduce试用范围

  • 要想使用mapreduce首先要确保输入可以相对独立的进行计算(map),数据之间没有计算依赖关系

疑问

  1. 一个mapreduce中,reduce是否需要等待所有mapper执行后才执行?
    需要,在 mapper 结束之前，reducer 很难知道属于一个 key 的数据是否收集完整，因此如果过早地开始 reduce，无法保值结果的正确性

  2. 如果一个mapper执行特别慢会拖慢整个任务(长尾现象)
    为了应对长尾现象（一个特别慢的子任务拖慢整个任务），MapReduce提供了 Backup Task的机制：当一个MapReduce接近结束时，master 对还处理 in-progress状态的task额外的调度备份执行，当primary和backup中一个执行成功就标记成功。

  3. Mapreduce中Map与Reduce任务的个数如何确定
    Mapreduce中Map与Reduce任务的个数

参考

Google-MapReduce中文版
Google-MapReduc
论文笔记：MapReduce
MapReduce论文笔记
MapReduce之Shuffle过程详述
https://blog.csdn.net/zhanglh046/article/details/78360762
https://github.com/feixiao/Distributed-Systems

发表于 | 分类于
字数统计: | 阅读时长 ≈

什么是守护进程

简单说,守护进程是在后台运行不受终端控制的进程

守护进程与孤儿进程区别联系

  • 孤儿进程是因为父进程异常结束了，然后被1号进程init收养。守护进程是创建守护进程时有意把父进程结束，然后被1号进程init收养,
  • 虽然他们都会被init进程收养，但是他们是不一样的进程。守护进程会随着系统的启动默默地在后台运行，周期地完成某些任务或者等待某个事件的发生，直到系统关闭守护进程才会结束。孤儿进程则不是，孤儿进程会因为完成使命后结束运行。

发表于 | 分类于
字数统计: | 阅读时长 ≈

Hadoop的MapReduce是对MapReduce的工程实践,做了大量的优化

  1. MapReduce中排序发生在哪几个阶段？？这些排序是否可以避免，为什么？？
    一个MapReduce作业由Map阶段和Reduce阶段两部分组成，这两阶段会对数据排序,
  • Map Task会在本地磁盘输出一个按照key排序（采用的是快速排序）的文件( 实际上Map阶段的排序就是为了减轻Reduce端排序负载)
  • 在Reduce阶段，每个Reduce Task会对收到的数据排序,
  1. 如何排序,用什么排序算法

发表于 | 分类于
字数统计: | 阅读时长 ≈

常见