Data Parallelism
in data parallelism : master gpu - weights copied to slave gpus - fwd pass done on part of data - gradients sent back to master, summed and master model updated. - so for every feed-forward, every model on every slave device is essentially destroyed, makes it slow.
Parallelism
here different layers of a model go to different gpus.
direct set prediction
what is direct set prediction problem?
entence embed-ding
sentence embedding. not word embedding
feature-based, parameter-based, andinstance-based MTSL models.
three types of multi task models are there.
For example, similar to MTL,transfer learning also aims to transfer knowledgefrom one task to another but the difference lies inthat transfer learning hopes to use one or moretasks to help a target task while MTL uses multi-ple tasks to help each other.
what is the difference between transfer learning and multi task learning.
from different instrumentsand proceeding to sounds from the same instrument
somewhat similar to ohem.
he gradients can point to more or less randomdirections
Methods of evaluating similarity ty
conventional wisdom has been that there is a tradeoffbetween image quality and variation, but that view has been recently challenged
either you can make image clearer or you can vary it. This was the conventional wisdom.
PROGRESSIVEGROWING OFGANS FORIMPROVEDQUALITY, STABILITY,ANDVARIATION
Annotated by @rajatmodi62
primarilyuse the improved Wasserstein loss, but also experiment with least-squares loss.
Need to understand what is wassertein loss. Will implement both of these.
report our result
hello
Towards a good design of mask information flow, we firstrecall the design of the cascaded box branches in CascadeR-CNN
They have improved the information flow in the network of multiple masksts
f regularization in order to utilize the regularization from largelearning rates and gain the other benefit - faster training
If we use large learning rates, we cannot use that much regularization.
learning rate test-> plot a graph of test accuracy with learning rate. Ideally, the learning rate appears to overfit or underfit. Training only starts if the learning rate is kept to be small. We can increase it later.
If large region in Lr test is flat, we can take that range of learning rates and obtain super convergence
ur first example is with a shallow
Takeaway , plot the test loss for multiple learning rates. Lower learning rate will show that test loss will continue to decrease. It does not increase anywhere so underfitting.
Increase the lr then , and see that initial falling is there, after which the accuracy does not improve.
f large learning rate and small batch size
old papers kept large learning rates and small batch size
Object detection performance, as measured on thecanonical PASCAL VOC dataset, has plateaued in the lastfew years. The best-performing methods are complex en-semble systems that typically combine multiple low-levelimage features with high-level context. In this paper, wepropose a simple and scalable detection algorithm that im-proves mean average precision (mAP) by more than 30%relative to the previous best result on VOC 2012—achievinga mAP of 53.3%. Our approach combines two key insights:(1) one can apply high-capacity convolutional neural net-works (CNNs) to bottom-up region proposals in order tolocalize and segment objects and (2) when labeled trainingdata is scarce, supervised pre-training for an auxiliary task,followed by domain-specific fine-tuning, yields a significantperformance boost. Since we combine region proposalswith CNNs, we call our methodR-CNN:Regions with CNNfeatures. We also compare R-CNN to OverFeat, a recentlyproposed sliding-window detector based on a similar CNNarchitecture. We find that R-CNN outperforms OverFeatby a large margin on the 200-class ILSVRC2013 detectiondataset. Source code for the complete system is available a
Old R-CNN, pipeline region extraction->warp in a single image->feed to cnn->o/p is bounding box coordinate and class,classification is done by passing to a family of svm trained on a classification task .
Needs 2000 regions, slow since the proposals need to be deesgined for each of the region
Alexandre Alahi
They don't rely on image optimization for style transfer. Instead they are training feed forward networks to apply style to an input image. I need to look at the code to understand can they apply the style in real time. _> in a single forward pass. The loss comes from the vgg model , maybe they use generator- discrominator architecture for the dcgan work , check it later on (supplementary material)
Then what is hte number of iterations which are present in the table.
perceptualloss functions
loss functions from high features
per-pixelloss between the output and ground-truth images
Image annotation methods are used
ons along their edges. This might originate from the zero-padding that is usedfor the convolutions in the VGG network and it could be interesting to investigate the effect of such padding onlearning and object recognition performance
[Future Idea]
Zero Padding can impact object recognition performance. Some neurons get trained to look at the zero paddinf and ht eimage edge,
non-texture image taken from the ImageNet validation set [23] (Fig. 2, last column). Our algorithmproduces a texturised version of the image that preserves local spatial information but discards theglobal spatial arrangement of the image. The size of the regions in which spatial information ispreserved increases with the number of layers used for texture generation. This property can beexplained by the increasing receptive field sizes of the units over the layers of the deep convolutionalneural network
A girl is there, she is repeated across entire image. In first layer output many girls repeated, ideally in the end layer it should represent just one girl in the image. In CNN, at higher layers, only filters remain , size of the activations reduce. Take the motivation from there to understand the result.
Figure 2: Generated stimuli. Each row corresponds to a different processing stage in the network.
[Look at the texture information] Texture refers to the localization of thepattern, ideally the pattern should not be repeated across the entire image. That's why they multiplied the feature activations, to lose the spatial information .
[In the results], the outputs of lower layers repeat the patterns throughout the image. In the higher layer output , the locality of the pattern is there. Only once unique girl is seen, image information appears to be reflected properly.
max-pooling operation by average pooling improved the gradient flow and one obtains slightly cleanerresults, which is why the images shown below were generated with average pooling
[Max pool->Average Pool] Layers replaced to get better statistics,
Design Inspiration from Generative Networks
This is a test annotation
If pneuomothorax is not there , then none of the patch cntains the disease Otherwise, one of the patch at least has the disesase , and the best patch is tried to be found out.
-Disadvantage- MIL only offers the rectangular patches information as of now .
CNN
CNN are good for classification problems. When local information is needed, like where exactly the pneumothorax is, they face a problem.
Image Classification
basic
A Neural Algorithm of Artistic Style
Uses CNN. formulates a style function from later layers of the cnn , and the overall image function from the earlier layers of the cnn
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Completed CycleGAN , use lateron
Style Transfer
Collection of the Neural Style Transfer Papers
Consistent Individualized Featur
checking the annotation tool in the check box