More on Haystack

Haystack powers allyoucanupload and will soon power all of Webshots (Webshots is a photo sharing community with 19,000,000 members who have uploaded over 375,000,000 photos).  Allyoucanupload is an image hosting service that we built to run alongside Webshots.

Haystack is designed to provide a very scalable, reliable and cost effective platform for object storage and delivery to the Internet.  It just went live 2 weeks ago - we are currently using it for the allyoucanupload Webshots image hosting service (gif, jpeg and png).   In the very near future, it will serve all Webshots photos, and soon, video. 

I'm doing a long post because I'm very proud of what our technical team has accomplished, but also to give some insight into how finance, user and technical strategy intersect in our group.  Building a large and sustainable (aka profitable) business in social media requires a balancing act between the delivering on a great user promise, a revenue model and keeping your costs under control.  Haystack will allow us to deliver very robust storage solutions for users at a very low marginal cost.

Disclaimer/Credit: Almost all of the technical content in this blog post was written by Paul O, who runs CNET Networks' data center services, including database architecture, network systems and operations and the actual data center.  Paul, Jim, Rodolphe, Marcus, and Matthew built Haystack - they don't (yet) have blogs so I'm doing this post.  Please do not attribute any technical props to me because of this post.  My only contribution to Haystack was to approve its development and cheerlead along the way.

Haystacks' content (social media files) has several interesting characteristics: it grows without bound; it tends toward write-once, read-many; the most recent content tends to be the most frequently accessed.  Haystack's design leverage these characteristics. 

The challenge: A big financial and therefore technical issue is the relationship of storage to delivery.  It's relatively easy to deliver a small number of files to a lot of people.  And a large number of files to a small number of people.  A large number of files to a large number of people gets more complicated and can get very expensive very quickly.

Haystack gives us the ability to finely match the raw storage capacity of the system with it's overall IO capacity.  Haystack grows very naturally through incremental addition of capacity.  Haystack is designed to handle failures automatically and to keep reliability constant as the system grows.  Haystack uses commodity hardware and software.

The promise of Haystack is that we can handle reliability at scale at a very low (perhaps the lowest) marginal operating cost. Reliability means that we never have to say we're sorry - we lost your photos.  Scale is scale.  Low marginal cost directly goes to our ability to give users as much storage as we can, and run the least intrusive ads, while running Webshots with a sustainable profit margin - and keeping the data center team focused on talent vs hands and well paid:)  You'll see the effects of Haystack on Webshots in our upcoming redesign and soon to be revised storage limits. 

BSU  Haystack consists of many Basic Storage Units (BSU), which are just servers with a lot of disks.  Content is scattered more or less randomly across all BSU and spindles to maximize the IO throughput of the system.  Multiple copies of the content are maintained on disparate equipment so that no single failure can loose all copies of an object.

Failure: In the event of a component failure, Haystack immediately begins a process to copy the "missing" content from one of the redundant sources to the available components.  Because the content is scattered across all available components, recovery time is on the order of 1/N.  Of course, the failure rate is on the order of N, so the overall availability is on the order of N * 1/N or a constant.  Recovery is also has very little impact on the overall performance of the system.

As new capacity is added, existing content is migrated to the new capacity to rebalance the storage and IO across all available units.  The rebalancing time is approximately constant.

Separation of Church and State
To minimize the overhead in tracking the location of any given object, Haystack puts objects into buckets and needs to track only the location of the buckets.  The applications using Haystack must independently track metadata about each object including it's bucket.  At some level, a bucket is really just a directory and each BSU knows what buckets it contains.  Haystack maintains a proper cache of the bucket locations and each BSU checks in periodically to report the state of its buckets.  The proper cache can easily be rebuilt and the overall system is very tolerant of data inconsistencies between clients, the cache and the BSUs.

Processes
Various processes monitor the overall condition of Haystack and initiate actions as needed to maintain the health.  For example, when new capacity is added, these monitoring processes detect the availability of "under-utilized" capacity and begin a bucket migration process to bring the new capacity up to the same levels as the old capacity.

Some of the jobs have names...they are:
leon's job is to identify and remove extra instances
jeopardy makes more copies of data that has too few copies
scalpel removes outdated copies of data
optimist moves data to fill up less-full nodes
pessimist moves data from busy nodes to less busy nodes

Changing the drives over time
Currently we use 400 GB sata drives.  As time goes by, the average age of the content will grow and the average number of access per object will decrease.  This will allow us to introduce larger capacity disk drives as the system grows, helping to keep the overall hosting and depreciations costs low.  The number of sata drives attached to a given BSU is determined by the overall network throughput of the BSU and the ability of the BSU to effectively use the file system cache.

Caching
Haystack uses various content caching and redirection mechanisms to both hide the complexity of the system from clients and to leverage the raw IO capacity of the spindles.  The system is very loosely coupled and designed to be quite tolerant of failures.  This means that failures are very localized.  In addition, the types of failures that can have the largest negative impact are with very proven technologies, such as file systems and disk arrays.  Thus the overall reliability is high and the operational costs are low.

Rant - lots of private social media properties are venture funded and lose tons of money storing/serving content to get big to either get bought or change the model. 

Finale The challenge at public companies like CNET Networks is to build a social media business model as well as a user & technical model from the "get go".  That operating plan must be good for user and cheap enough to operate so we can return a healthly return on invested capital to our shareholders.  Haystack is a significant innovation that should help us to build a better product for users, scale for marketing partners, and generate operating cash flow for shareholders.

Whew - if you're still reading then you are likely one of the Webshots or CNET Networks engineers - nice work folks :)

Comments

[asdf]

world of warcraft power leveling wow power leveling power leveling wow gold 福彩3D 直流电源 paper cup machine Mass air flow 程控交换机 餐饮软件 餐饮软件 automatic rigid box line paper converting and wrapping machines automatic box makers rigid set-up gift box,paperboard converting equipment

气动马达 气动搅拌机 防爆马达 防爆搅拌机 气动搅拌机 空气马达 防爆马达 气动马达 油漆搅拌机 油漆搅拌器 气动搅拌器 防爆搅拌器 油墨搅拌器 油墨搅拌机 安检门 金属探测器 金属探测仪 thermoforming Equipment bag making machine 工业设计 产品设计 储罐 中药提取设备 乳化机 反应釜 真空干燥箱 厚壁钢管

power leveling wow power leveling paper box paper box equipment thermoforming Equipment Plastic Machinery Plastic Machine 马达 气动马达 搅拌机

[mirc]

www.sohbet24.org/konya-chat.htm

www.konyachatsohbet.com

www.konyapartner.net

[mirc]

http://www.sohbet24.org/konya-chat.htm

http://www.konyachatsohbet.com

http://www.konyapartner.net

[fengyun]

When the wolf [url=http://www.game4power.com/]wow gold [/url] finally found the hole [url=http://www.game4power.com]Buy Wow Gold[/url] in the chimney he [url=http://www.wowgoldone.com/]Cheap WoW Gold[/url] crawled down and KERSPLASH right into that kettle of water [url=http://www.wowgoldone.com/]cheapest wow gold[/url] and that was the end of his troubles with the big bad wolf.

[url=http://www.game4power.com/]game4power[/url],[url=http://www.game4power.com/buy-gold/]buy cheap wow gold [/url]

The next day the [url=http://www.game4power.com/buy-gold/]wow gold cheap[/url] little pig invited his mother over . She said "You see it is just as [url=http://www.game4power.com/]buy gold wow[/url]I told you. The way to [url=http://www.
itemchannel.com]Wow Gold[/url]get along in the world is to do things as well as you can." Fortunately for that [url=http://www.itemchannel.com]world of warcraft gold [/url] little pig, he [url=http://www.game4power.com/buy-gold]Cheapest wow Gold[/url]learned that lesson. And he just lived happily ever after!

[konya chat]

hey thanks

konya chat

[asdf]

world of warcraft power leveling wow power leveling power leveling wow gold 橡塑发泡 直流电源 paper cup machine Mass air flow 程控交换机 餐饮软件 餐饮软件 收银机 automatic rigid box line paper converting and wrapping machines automatic box makers rigid set-up gift box,paperboard converting equipment

气动马达 气动搅拌机 防爆马达 防爆搅拌机 气动搅拌机 空气马达 防爆马达 气动马达 油漆搅拌机 油漆搅拌器 气动搅拌器 防爆搅拌器 油墨搅拌器 油墨搅拌机 安检门 金属探测器 金属探测仪 thermoforming Equipment bag making machine 工业设计 产品设计 储罐 中药提取设备 乳化机 反应釜 真空干燥箱 厚壁钢管

power leveling wow power leveling paper box paper box equipment thermoforming Equipment Plastic Machinery Plastic Machine 马达 气动马达 搅拌机

asdf

[jewelry store]

http://www.jewelry-store-wholesaler.com

[gw gold]

It is really a good post.

[mobilya]

thanks beautiful

[pvpvip]

6.11chttp://www.uswotlk.com/ world of warcraft gold
http://www.pvpvip.com/

[fengyun]

When the wolf [url=http://www.game4power.com/]wow gold [/url] finally found the hole [url=http://www.game4power.com]Buy Wow Gold[/url] in the chimney he [url=http://www.wowgoldone.com/]Cheap WoW Gold[/url] crawled down and KERSPLASH right into that kettle of water [url=http://www.wowgoldone.com/]cheapest wow gold[/url] and that was the end of his troubles with the big bad wolf.

[url=http://www.game4power.com/]game4power[/url],[url=http://www.game4power.com/buy-gold/]buy cheap wow gold [/url]

The next day the [url=http://www.game4power.com/buy-gold/]wow gold cheap[/url] little pig invited his mother over . She said "You see it is just as [url=http://www.game4power.com/]buy gold wow[/url]I told you. The way to [url=http://www.
itemchannel.com]Wow Gold[/url]get along in the world is to do things as well as you can." Fortunately for that [url=http://www.itemchannel.com]world of warcraft gold [/url] little pig, he [url=http://www.game4power.com/buy-gold]Cheapest wow Gold[/url]learned that lesson. And he just lived happily ever after!

[HD LCD monitor]

It's nice, I am learning, thank you

[fengyun]

When the wolf [url=http://www.game4power.com/]wow gold [/url] finally found the hole [url=http://www.game4power.com]Buy Wow Gold[/url] in the chimney he [url=http://www.wowgoldone.com/]Cheap WoW Gold[/url] crawled down and KERSPLASH right into that kettle of water [url=http://www.wowgoldone.com/]cheapest wow gold[/url] and that was the end of his troubles with the big bad wolf.

[url=http://www.game4power.com/]game4power[/url],[url=http://www.game4power.com/buy-gold/]buy cheap wow gold [/url]

The next day the [url=http://www.game4power.com/buy-gold/]wow gold cheap[/url] little pig invited his mother over . She said "You see it is just as [url=http://www.game4power.com/]buy gold wow[/url]I told you. The way to [url=http://www.
itemchannel.com]Wow Gold[/url]get along in the world is to do things as well as you can." Fortunately for that [url=http://www.itemchannel.com]world of warcraft gold [/url] little pig, he [url=http://www.game4power.com/buy-gold]Cheapest wow Gold[/url]learned that lesson. And he just lived happily ever after!

[HD LCD monitor]

Thanks for sharing this great article!

[apri]

Is the quality of service, the price is amazing, as soon as possible mmoinn to buy wow gold it!

[sare]

gf u7

[zenghui]

People taking the [URL=http://www.ghdsale.com]ghd[/URL]stairs instead of elevator at work can expect to live longer, according to a Swiss study released[URL=http://www.ghdsale.com]hair straighteners[/URL] on Monday.
Regularly walking from[URL=http://www.ghdsale.com]ghd uk[/URL] floor to floor in an office building decreased mortality[URL=http://www.ghdsale.com]ghd straighteners[/URL] risk by 15 percent, said Dr.[URL=http://www.ghdsale.com]ghd hair styler[/URL] Philippe Meyer, the main author of the study, which was [URL=http://www.ghdsale.com]ghd hair straighteners[/URL]done at the University Hospital of Geneva.
Banning the use of lifts[URL=http://www.edhardysale.net]ed hardy[/URL] and escalators led to better fitness, less body fat,[URL=http://www.edhardysale.net]ed hardy clothing[/URL] trimmer waistlines and a drop in blood pressure,[URL=http://www.edhardysale.net]ed hardy caps[/URL] the study found.
Using the stairs improves fitness, [URL=http://www.edhardysale.net]ed hardy discounts[/URL]body composition, blood pressure and lipid profiles, Meyer was quoted as[URL=http://www.edhardysale.net]ed hardy online[/URL] saying by the Swissinfo news website.
"The challenge remains to[URL=http://www.edhardysale.net]ed hardy tshirts[/URL] develop successful population-based [URL=http://www.edhardysale.net]ed hardy women[/URL]interventions, which promote physical activities that can be [URL=http://www.ghdbeauty.com]Cheap GHD[/URL]easily integrated into everyday life,"[URL=http://www.tiffanyhotsale.com/key-ring-c-10_19.html]Tiffany Key Ring[/URL] he said.
For the study, 77 [URL=http://www.ghdbeauty.com/ghd-pure-c-70.html]GHD pure[/URL]employees from Geneva University with a sedentary [URL=http://www.ghdbeauty.com/ghd-kiss-c-69.html]GHD kiss[/URL]lifestyle were recruited to take only the stairs over[URL=http://www.ghdbeauty.com/ghd-mk4-black-c-66.html]GHD MK4 Black[/URL] a three-month period, Swissinfo reported.
Results showed an increase in [URL=http://www.ghdbeauty.com/ghd-mk4-pink-c-67.html]GHD MK4 Pink[/URL]aerobic capacity, a[URL=http://www.tiffanyhotsale.com/brooch-c-10_21.html]Tiffany Brooch[/URL] decline in waist circumference, [URL=http://www.tiffanyhotsale.com/cuff-button-c-10_20.html]Tiffany Cuff Links[/URL]weight, fat mass, blood pressure[URL=http://www.ghdbeauty.com/ghd-mk4-gold-c-68.html]GHD MK4 Gold[/URL] and cholesterol.
"This suggests that stair[URL=http://www.ghdbeauty.com/ghd-purple-c-71.html]GHD Purple[/URL] climbing can have major public health implications." [URL=http://www.tiffanyhotsale.com]Tiffany Necklaces[/URL]Meyer said.
Professor Adam Timmis, consultant[URL=http://www.tiffanyhotsale.com]Tiffany Ring[/URL] cardiologist at The London NHS [URL=http://www.tiffanyhotsale.com]Tiffany Bangle[/URL]Trust, said: "It's a small study but valuable[URL=http://www.tiffanyhotsale.com]Discount Tiffany[/URL] because it provides a practical way for[URL=http://www.tiffanyhotsale.com/tiffany-1837-c-12_38.html]Tiffany 1837[/URL] busy working people to increase their exercise[URL=http://www.tiffanyhotsale.com/tiffany-atlas-c-12_39.html]Tiffany Atlas[/URL] capacity.
"Although the amount of [URL=http://www.tiffanyhotsale.com/watches-c-10_15.html]Tiffany Watch[/URL]exercise appears small, the benefits[URL=http://www.tiffanyhotsale.com/tiffany-paloma-picasso-c-16_27.html]Tiffany Paloma Picasso[/URL] were clear in improving physical fitness and reducing body fat and blood pressure."

[mırc]

very nice admin thank you

[female viagra]

mirc and ve nice web

[islami sohbet]

http://www.sefaat.net
http://www.islamisohbetim.tk
http://www.ilahilerezgiler.com

[Viagra Online]

Can you give more info about Haystack ?

[kangal]

we are thanking for the knowledge

[speles]

Hi, nice share, thanks.

[konya chat]

thanks of

[konya chat]

thanks of