28 Matching Annotations
- Nov 2021
-
lilianweng.github.io lilianweng.github.io
-
Three techniques to avoid losing critical information at half-precision: Full-precision master copy of weights. Maintain a full precision (FP32) copy of model weights that accumulates gradients. The numbers are rounded up to half-precision for forward & backward passes. The motivation is that each gradient update (i.e. gradient times the learning rate) might be too small to be fully contained within the FP16 range (i.e. 2−242−242^{-24} becomes zero in FP16). Loss scaling. Scale up the loss to better handle gradients with small magnitudes (See Fig. 16). Scaling up the gradients helps shift them to occupy a larger section towards the right section (containing larger values) of the representable range, preserving values that are otherwise lost. Arithmetic precision. For common network arithmetic (e.g. vector dot-product, reduction by summing up vector elements), we can accumulate the partial results in FP32 and then save the final output as FP16 before saving into memory. Point-wise operations can be executed in either FP16 or FP32.
混合精度中是通过哪些方式保证精度不会损失的？
-
two major memory consumption of large model training: The majority is occupied by model states, including optimizer states (e.g. Adam momentums and variances), gradients and parameters. Mixed-precision training demands a lot of memory since the optimizer needs to keep a copy of FP32 parameters and other optimizer states, besides the FP16 version. The remaining is consumed by activations, temporary buffers and unusable fragmented memory (named residual states in the paper).
深度网络训练中的显存开销主要是哪些？
-
- May 2021
-
plausible.io plausible.io
-
Hooray!
Tags
Annotators
URL
-
- Mar 2021
-
ethanmarcotte.com ethanmarcotte.com
- Jul 2020
-
amp.dev amp.dev
-
One of the most important concepts in AMP’s design is its focus on reducing the amount of DOM reflow required to render its web pages. To reduce DOM reflow, AMP includes a layout system to ensure the layout of the page is known as early as possible in the lifecycle of downloading and rendering the page.
-
While stylesheets can be reworked relatively easily with AMP by inlining the CSS, the same is not true for JavaScript. The tag 'script' is disallowed except in specific forms. In general, scripts in AMP are only allowed if they follow two major requirements: All JavaScript must be asynchronous (i.e., include the async attribute in the script tag). The JavaScript is for the AMP library and for any AMP components on the page. This effectively rules out the use of all user-generated/third-party JavaScript in AMP except as noted below.
-
The problem is that this is an external stylesheet reference. In AMP, to keep the load times of documents as fast as possible, you cannot include external stylesheets.
-
The meta charset information must also be the first child of the <head> tag. The reason this tag must be first is to avoid re-interpreting content that was added before the meta charset tag.
But what if another tag also specified that it had to be the first child "because ..."? Maybe that hasn't happened yet, but it could and then you'd have to decide which one truly was more important to put first? (Hopefully/probably it wouldn't even matter that much.)
-
-
amp.dev amp.dev
-
-
If you can, use AMP like any other JavaScript library to build your site and forget about the canonical linking. Using AMP to build a whole website dramatically reduces your maintenance burden.
-
-
amp.dev amp.dev
-
You don't have to name your AMP files as .amp.html. In fact, AMP files can have any extension you want. It's common to see publishers differentiating AMP pages from their canonical versions by using parameters in the url. For example: http://publisher.com/article.html?amp.
-
-
amp.dev amp.dev
-
Building blocks that ensure performance It takes a lot of time and effort to build a great website. AMP components are already optimized for the best performance.
-
Used by global platforms like Google, Bing and Twitter, AMP allows users a native-feeling experience across all platforms by defaulting to AMP pages when available.
-
-
wordpress.org wordpress.orgAMP1
-
AMP formerly stood for “Accelerated Mobile Pages”, but now works completely across desktop and mobile.
Tags
Annotators
URL
-
- Dec 2019
-
knightcolumbia.org knightcolumbia.org
-
And other recent developments suggest that doing so could overcome many of the earlier pitfalls of protocol-based systems, potentially creating the best of all words: useful internet services, with competition driving innovation, not controlled solely by giant corporations, but financially sustainable, providing end users with more control over their own data and privacy—and providing mis- and disinformation far fewer opportunities to wreak havoc.
Some of the issue with this then becomes: "Who exactly creates these standards?" We already have issues with mega-corporations like Google wielding out sized influence in the ability to create new standards like Schema.org or AMP.
Who is to say they don't tacitly design their standards to directly (and only) benefit themselves?
-
- Sep 2018
-
www.ampproject.org www.ampproject.org
Tags
Annotators
URL
-
- Mar 2018
-
-
We worked with the industry to launch the open-source Accelerated Mobile Pages Project to improve the mobile web
There was some collaborative outreach, but AMP is really a Google-driven spec without significant outside input.
See also: http://ampletter.org/
-
With AMP Stories, which is now in beta, publishers can combine the speed of AMP with the rich, immersive storytelling of the open web.
"With AMP Stories, which is now in beta, publishers can combine the speed of AMP with the rich, immersive storytelling of the open web."
Is this sentence's structure explicitly saying that AMP is not "open web"?!
-
- Nov 2017
-
www.nytimes.com www.nytimes.com
-
Whistle-blowers and dissidents might need to use a different platform.)
The way that he casually mentions whistle blowers and dissidents is troubling to say the least. Also, will dig up studies, but removing anonymity hasn't really shown to decrease trolling or other bad behavior. Also, "privacy" anyone?
http://theweek.com/articles/632929/problem-internet-trolls-isnt-anonymity
-
-
-
GOOG promotes lock-in and proprietary technologies such as Firebase and Google-dependent AMP installations as much as it advocates open PWAs. GOOG does not consistently defend the open Web. They dropped XMPP in Gtalk, and Gtalk itself was deprecated, favoring Google Hangouts with a proprietary protocol. Chrome Web Store is a walled garden like App Store. They shutdown Google Reader based on RSS, an open standard. Google Cloud TPU is proprietary hardware that only exists in their datacenters, supporting their open source framework TensorFlow. Google Inbox suffers “proprietary creep”: non-standard, closed algorithms that promise to organize your life, an essential component of a lock-in based business model.
Tags
Annotators
URL
-
- Jul 2017
-
www.youtube.com www.youtube.com
-
How To Add AMP To Your WordPress Website - Accelerated Mobile Pages
Tags
Annotators
URL
-
-
www.youtube.com www.youtube.com
-
How to Properly Setup Google AMP on Your WordPress Site
How to setup google amp on wordpress
-
-
www.ampproject.org www.ampproject.org
-
AMP provides a great user experience across many platforms
Description about google acelerated mobile platform
Tags
Annotators
URL
-