CS698N - Recent Advances in Computer Vision

Instructor: Gaurav Sharma, Assistant Professor, CSE, IITK
Contact: grv AT cse.iitk.ac.in ([CS698N] in email subject; o/w ignored)
Lectures: Wed and Fri, 10h30–12h00
Office hours: Wed 15h00–16h00 or by appointment
TA: Saptarshi Gan (sapgan AT iitk.ac.in)

Summary

In this course, we will look at a subset of topics in the following exciting sub-areas of research in Computer Vision. This list of topic is adaptable depending on the level and interests of the students actually taking the course.

• Human Analysis eg. actions, pose estimation, facial analysis, attribute recognition, pedestrian detection
• Language and Vision eg. image captioning, visual question answering
• Image segmentation eg. semantic segmentation and multi resolution edge estimation, instance segmentation

There will be a significant project component -- you are expected to mainly learn by doing.

I will also try to organize ~4 guest lectures (probably) over video conferencing where researchers who are actually working on the forefront of a researh problem will present their recent work.

Department's anti-cheating policy is applicable on your participation in the course.

Grading

• Project (total 50%)
  
  
  • State-of-the-art presentation 10%
  • Proposal 10%
  • Mid-sem progress presentation 10%
  • Final report and presentation/demo 20%


• Assignments (total 30%)
  
  
  • 1 page extended abstract of a research paper 10%
  • Review a research paper 10%
  • Seminar presentation 10%


• End-sem 20%

Announcements

• Final project submissions are due on the 07th November 2016, 23:59h IST
• Assignment 3 is due on the 31st October 2016, 23:59h IST
• Assignment 2 is due on the 21st October 2016, 23:59h IST
• Assignment 1 is due on the 19th 20th August 2016, 23:59h IST
• Dear auditors please send me an email to add your email id to the course related emails.

Guest Lectures

The following people have kindly agreed to give guest lectures. The lecture schedules are to be decided (unless given). Scroll to the end of the page for details about the talks which have been scheduled so far.

• Chetan Arora, IIIT Delhi. Topic: Activity Recognition in First Person Videos [details]
• Hakan Bilen, University of Oxford. Topic: Weakly supervised object detection [details]
• Makarand Tapaswi, University of Toronto. Topic: Understanding Stories by Joint Analysis of Language and Vision [details]
• Omkar Parkhi, Zoox Inc., Topic: Face recognition in still images and videos [details]
• Jan Hosang, Max Planck Institute for Informatics. Topic: Detection Proposals and Learning Non-Maximum Suppression [details]

Assignments

The first two Assignments are to be done in groups of 2-3 people. Only one file per group needs to be uploaded by any one of the group members. Please use the IITK CSE moodle website for submitting the assignments. The time stamp of the moodle upload will be considered your submission time.
Any submission by mail will not be considered.

1. Pick a paper (within the areas given in Summary above) which catches your attention from a recent computer vision conference (ECCV 2016, ICCV 2015, CVPR 2015 or 2016 -- not workshop papers) and write a one page extended abstract in BMVC 2015 Latex template, section on "Instructions for submission of camera ready papers" (in groups of 2-3; need not be same as project group). The idea here is to understand the paper and highlight the contributions (problem solved, novelty and other advantages of the approach and experimental results) made.

   The proceedings for recent ICCVs and CVPRs are open access. For ECCV 2016 (which will happen in Sep 2016 but the acceptance decisions are out) you will have to search the internet or look at webpages of some prominent researchers or search arXiv (1. the X is read as chi -- so arXiv is read as "Archive", 2. keep looking after every few days as new papers keep on coming).

   Some examples -- object detection, deep face recognition, and many more at BMVC 2015 website.

   Due 19th 20th August 2016, 23:59h IST

2. Pick a paper (within the areas given in Summary above) which catches your attention from a recent computer vision conference (ECCV 2015, 2016; ICCV 2015 or CVPR 2015, 2016 -- not workshop papers)) and write a review for the paper (in groups of 2-3; need not be same as project group). In particular, you should
   
   

   • Summarize the paper
   • Point out the strengths of the paper
   • Point out the weaknesses of the paper
   • Say if you would like to accept it or reject it with justifications
   • Optional -- Contrast and compare it with other recent papers/methods
   
   For reference, reviewer guidelines for CVPR 2015 are here. Also, you can find numerous articles by different academics on how to review papers. I will put up some light formatting requirements soon.
   Due 21st October 2016, 23:59h IST

3. One ECCV 2016 paper each has been assigned to your group (see the mailing list). You have to prepare a presentation for explaining the paper in sufficient details in 15 minutes. You are also expected to read some of the other papers and contribute to class discussions. There will be time for questions in addition to the 15 min presentation time.
   Due 31st October 2016, 23:59h IST

Late Submission Policy

You have total 4 late submission days (for the assignments, the project proposal and the project end-sem reports). After that, you will be penalized by deducting 5 marks for each late day (over the 4 late days allowed). The late days cumulate over the different deadlines, i.e. if you were late by 1 day for assignment 1 and then late again by 3 days for assignment 2, you would have used up the total of 4 late days quota. The project presentations (state-of-the-art, mid-sem and end-sem) have to be uploaded two days prior to the presentation day (e.g. if presentation is on the 5th, the deadline for uploading presentations would be 23:59h of the 3rd) and this is a hard deadline - if you fail to do so you will not be allowed to give the respective presentation. The idea here is that you should practice giving the presentation at least a day in advance. All the deadlines will be announced on the webpage in due time.

Project Details

The project will be a significant component of the course and will be continuously evaluated. The project should not overlap with any project, e.g. for any other course, that you might have done previously or are doing currently. If there is any such overlap, it should be decalred clearly; failure to do so will be considered as cheating and the Department's anti-cheating policy will be used to deal with such cases.

You are expected to choose a problem for your project, implement/reproduce approximately results from an existing paper using open-source or available code and finally either (i) implement a key algorithm in that yourself or (ii) go beyond that work by identifying some weakness and improving on it.

Milestones for project; see the course calendar below for exact deadlines.

• Make groups of 2-3
• Proposal and State-of-the-art presentation - expectation is that you will read at least one paper on which you want to base your project on and submit a proposal by (i) describing the problem, (ii) discussing a few methods which try to solve the problem and (iii) your plan for the project towards solving that problem.
• Mid-sem project evaluation (presentation and short report) - reproduce some results using open-source libraries or available code (released by authors). Present or demo your results.
• End-sem project evaluation (presentation and report) - Implement one key algorithm yourself or go beyond the base method. Summarize your contribution and present or demo your results.

Only one file per group needs to be uploaded by any one of the group members. Please use the IITK CSE moodle website for submitting the project related presentations and report. The time stamp of the moodle upload will be considered your submission time.
Any submission by mail will not be considered.

Course Calendar

All deadlines below are at 23:59 IST of the respective dates.
The course material (slides etc.) are freely usable for educational and non-commercial research purpose, with due attribution. The material is avialable as is without any warranty, expressed or implied, whatsoever. Any commercial use requires prior written permission from the author. If you are the owner of any of the content included (eg. images), and feel that it has been unfairly used, kindly let me know and I will either attribute it to you as you specify or take it off, depending on your request.

Details of Guest Lectures

Speaker

Chetan Arora
Assistant Professor
Indraprastha Institute of Information Technology Delhi (IIIT Delhi)

Title

Activity Recognition in First Person Videos

Time and Venue

Date: 09 August 2016, Tuesday
Time: 1100–1200 (tea at 1045)
Venue: RM101, CSE
(In person)

Abstract

Wearable cameras like the GoPro are one of the best selling cameras these days. The always on nature and the first person point of view are the unique characetristics of such egocentric cameras giving access to the information not possible with traditional point and shoot cameras. Recognizing wearer’s activity is one of the core tasks in many egocentric applications. In this talk I will present some of our work in this area starting with our earlier work on long term activity recognition using traditional machine learning methods. I will then go on to explain how deep learning helped us to generalize the recognition to much larger class of activities for which designing hand tuned features was unthinkable.

Speaker Bio

Chetan Arora received his Bachelor’s degree in Electrical Engineering in 1999 and the Ph.D. degree in Computer Science in 2012, both from IIT Delhi. From 2000-2009 he was an entrepreneur involved in setting up companies working on various computer vision based products. From 2012 to 2014 he was post-doctoral researcher at Hebrew University, Israel. He is currently an Assistant Professor at IIIT Delhi. His broad areas of research include computer vision and image processing.

Speaker

Hakan Bilen
Postdoc
Visual Geometry Group
University of Oxford

Title

Weakly Supervised Object detection

Time and Venue

Date: 07 September 2016, Wednesday
Time: 1930–2030
Venue: KD101, CSE
(By video conferencing)

Abstract

Weakly supervised learning of object detection is an important problem in image understanding that still does not have a satisfactory solution. In this talk, we address this problem by improving different aspects of the standard multiple instance learning based object detection. We first present a method that can represent and exploit presence of multiple object instances in an image. Second we further improve this method by imposing similarity among objects of the same class. Finally we propose a weakly supervised deep detection architecture that can exploit the power of deep convolutional neural networks pre-trained on large-scale image-level classification tasks.

Speaker Bio

Hakan Bilen received his PhD degree in Electrical Engineering in 2013 and spent a year as a postdoctoral researcher at the University of Leuven in Belgium. He is currently a postdoctoral researcher in the University of Oxford since 2015. His research areas include computer vision and machine learning.

Speaker

Makarand Tapaswi
Postdoc
University of Toronto

Title

Understanding Stories by Joint Analysis of Language and vision

Time and Venue

Date: 21 September 2016, Wednesday
Time: 1400–1500 IST
Venue: RM101, CSE
(In person)

Abstract

Humans spend a large amount of time listening, watching, and reading stories. We argue that the ability to model, analyze, and create new stories is a stepping stone towards strong AI. We thus work on teaching AI to understand stories in films and TV series. To obtain a holistic view of the story, we align videos with novel sources of text such as plot synopses and books. Plots contain a summary of the core story and allow to obtain a high-level overview. On the contrary, books provide rich details about characters, scenes and interactions allowing to ground visual information in corresponding textual descriptions. We also work on testing machine understanding of stories by asking it to answer questions. To this end, we create a large benchmark dataset of almost 15,000 questions from 400 movies and explore its characteristics with several baselines.

Speaker Bio

Makarand Tapaswi received his undergraduate education from NITK Surathkal in Electronics and Communications Engineering. Thereafter he pursued an Erasmus Mundus Masters program in Information and Communication Technologies from UPC Barcelona and KIT Germany. He continued with the Computer Vision lab at Karlsruhe Institute of Technology in Germany and recently completed his PhD. He will be going to University of Toronto as a post-doctoral fellow starting in October.

Speaker

Omkar Parkhi
Researcher
Zoox Inc.

Title

Face recognition in still images and videos

Time and Venue

Date: 7 October 2016
Time: 1030–1200 IST
Venue: KD101, CSE
(By video conferencing)

Abstract

In this talk I will describes feature representations for face recognition, and their application to various activities relating to image and video datasets.

First, we will look at different “shallow” representations for faces in images and videos. The objective is to learn compact yet effective representations for describing faces. Specifically will see the effectiveness of “Fisher Vector” descriptors for this task. We show that these descriptors are perfectly suited for face representation tasks both in images as well as videos. I will also look at various approaches to effectively reduce their dimension while improving their performance further. These “Fisher Vector” features are also amenable to extreme compression and work equally well when compressed by over 2000 times as compared to their non compressed counterparts. These features have achieved the state-of-the-art results on challenging public benchmarks.

More recently the Convolution Neural Networks (CNN) have been dominating the field of face recognition as with the other fields of computer vision. Most of the public research on CNNs for face recognition has been contributed by the Internet giants like Facebook. At the same time, in the academic world, increasingly complex network architectures were introduced specifically for facial recognition. One such proposal used 200 trained networks for final score prediction. We aim to propose a simple yet effective solution to this problem and investigate the use of ``Very Deep’’ architectures for face representation tasks. For training these networks, we collected one of the largest annotated public datasets of celebrity faces requiring minimum manual annotations. We bring out specific details of these network architectures and their training objective functions essential to their performance and achieve state-of-art result on challenging datasets.

Having described these representation, I will explain their application to various problems in the field. We will look at a method for labeling faces in the challenging environment of broadcast videos using their associated textual data, such as subtitles and transcripts. We show that our CNN representation is well suited for this task and also propose a scheme to automatically differentiate the primary cast of a TV serial or movie from that of the background characters. We improve existing methods of collecting supervision from textual data and show that the careful alignment of video and textual data results in significant improvement in the amount of training data collected automatically, which has a direct positive impact on the performance of labeling mechanisms. We provide extensive evaluations on different benchmark datasets achieving, again, state-of-the-art results.

Further we show that both the shallow as well the deep features described above have excellent capabilities in switching modalities from photos to paintings and vice-a-versa. We propose a system to retrieve paintings for similar looking people given a picture and investigate the use of facial attributes for this task. Finally, I will show an “on-the-fly” real time search system that has been built to search through thousands of hours of video data starting from a text query. To ensure real time performance, we propose product quantization schemes for making face representations memory efficient. We also present the demo system based on this design for the British Broadcasting Corporation (BBC) to search through their archive.

All of these contributions have been designed with a keen eye on their application in the real world. As a result, most of discussed contributions have an associated code release and a working online demonstration.

Additionally I will also briefly describe some of our previous work on detecting deformable animals (cats and dogs) and their sub-categorization.

Speaker Bio

Omkar is an alumnus of CVIT, IIIT Hyderabad and did his PhD under Andrew Zisserman at Oxford. Currently he is with the autonomous driving startup Zoox Inc.

Speaker

Jan Hosang
PhD candidate
Max Planck Institute for Informatics

Title

Detection Proposals and Learning Non-Maximum Suppression

Time and Venue

Date: 26 October 2016
Time: 1930–2030 IST
Venue: KD101, CSE
(By video conferencing)

Abstract

The talk will focus on the very first and very last step in the common object detection pipeline. Proposals are a common technique to cut down the search space compared to typical sliding window detection, while keeping high detection quality. I will talk about the implications of the search space reduction and proposal evaluation.

Non-maximum suppression is a hand crafted post processing step that persists even though we like to think of object detectors as end-to-end trained systems. In its typical form it forces a trade-off between how many occluded objects can be detected and how many false detections are generated. I will present how it is possible to learn non-maximum suppression with a Convnet by posing it as rescoring task.

Speaker Bio

Jan Hosang received is Diploma in computer science at RWTH Aachen University in 2011. Since then he has interned at the handwriting recognition group at Google and joined the UdS Computer Science Graduate School in 2012. He is currently pursuing a PhD in computer science in the Computer Vision and Multi-modal Computing group at the Max Planck Institute for Informatics, Saarbrücken. His research interests are computer vision and machine learning, in particular object detection.