Sergi Caelles

I am a Senior Research Engineer in the Gemini team at Google DeepMind. Before, I was a PhD student at the Computer Vision Lab in Eidgenössische Technische Hochschule Zürich (ETH Zürich) under the supervision of Prof. Luc Van Gool and the guidance of Dr. Jordi Pont-Tuset. I received a Degree and M.Sc. in Telecommunications Engineering from the Universitat Politecnica de Catalunya (UPC).

I have interned at Bell Labs with Prof. Sebastian Randel, Facebook with Prof. Fernando de la Torre, and Google Research with Prof. Vittorio Ferrari. Throughout my career, I have worked in segmentation with special focus on video object segmentation by using deep learning. Currently, my focus is on advancing the video understanding capabilities of Gemini, see our latest progress update.

More info in my CV.

[Nov 2022]: VCT was accepted at NeurIPS 2022.

[Jun 2021]: I gave a talk in the 1st Future Video Conferencing Workshop at CVPR 2021.

[Jul 2020]: I joined the Perception team at Google Research as a Research Scientist.

[Jul 2019]: I am interning in the Perception team at Google Research until November 2019.

[Mar 2019]: We have published all the details for DAVIS 2019 Challenges.

[Aug 2018]: We have release all the object categories for DAVIS 2017.

[Jul 2018]: I am interning in the AR/VR team at Facebook until November 2018.

[Jun 2018]: The DAVIS Challenge 2018 has finished check the final results and publications.

[May 2018]: Our extension of OSVOS, OSVOS-S, has been accepted by TPAMI!

[Mar 2018]: Deep Extreme Cut (DEXTR) was accepted at CVPR 2018!

The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation

S. Caelles , J. Pont-Tuset , F. Perazzi , A. Montes , K.-K. Maninis , L. Van Gool

arXiv:1905.00737, 2019

[PDF] [Website] [BibTex]

@article{Caelles_arXiv_2019, 
  author = {S. Caelles and J. Pont-Tuset and and F. Perazzi and A. Montes and K.-K. Maninis and L. {Van Gool}},
  title = {The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation},
  journal = {arXiv:1905.00737},
  year = {2019}
}

Fast video object segmentation with Spatio-Temporal GANs

S. Caelles* , A. Pumarola* , F. Moreno-Noguer , A. Sanfeliu , L. Van Gool

arXiv:1903.12161, 2019

[PDF] [Website] [BibTex]

@inproceedings{Cae+19,
  author  = {S. Caelles and A. Pumarola and F. Moreno-Noguer and A. Sanfeliu and L. {Van Gool}},
  title   = {Fast video object segmentation with Spatio-Temporal GANs},
  booktitle = {arXiv:1903.12161},
  year    = {2019}
}

Deep Extreme Cut: From Extreme Points to Object Segmentation

K.-K. Maninis* , S. Caelles* , J. Pont-Tuset , L. Van Gool

Computer Vision and Pattern Recognition (CVPR), 2018

[PDF] [Website] [Code] [BibTex]

@inproceedings{Man+18,
  author  = {K.-K. Maninis and S. Caelles and J. Pont-Tuset and L. {Van Gool}},
  title   = {Deep Extreme Cut: From Extreme Points to Object Segmentation},
  booktitle = {Computer Vision and Pattern Recognition (CVPR)},
  year    = {2018}
}

The 2018 DAVIS Challenge on Video Object Segmentation

S. Caelles , A. Montes , K.-K. Maninis , Y. Chen , L. Van Gool , F. Perazzi , J. Pont-Tuset

arXiv:1803.00557, 2018

[PDF] [Website] [BibTex]

@article{Caelles_arXiv_2018, 
  author = {S. Caelles and A. Montes and K.-K. Maninis and Y. Chen and L. {Van Gool} and F. Perazzi and J. Pont-Tuset},
  title = {The 2018 DAVIS Challenge on Video Object Segmentation},
  journal = {arXiv:1803.00557},
  year = {2018}
}

Video Object Segmentation Without Temporal Information

K.-K. Maninis* , S. Caelles* , Y. Chen , J. Pont-Tuset , L. Leal-Taixé , D. Cremers , L. Van Gool

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2018

[PDF] [Website] [BibTex]

@Article{Man+18b,
   Author  = {K.-K. Maninis and S. Caelles and Y. Chen and J. Pont-Tuset and L. Leal-Taix\'e and D. Cremers and L. {Van Gool}},
   Title   = {Video Object Segmentation Without Temporal Information},
   Journal   = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
   Year    = {2018}
 }

Iterative Deep Learning for Road Topology Extraction

C. Ventura , J. Pont-Tuset , S. Caelles , K.-K. Maninis , L. Van Gool

British Machine Vision Conference (BMVC), 2018

[PDF] [Website] [Code] [BibTex]

@article{Venture_bmvc_2018, 
  author = {C. Ventura and J. Pont-Tuset and S. Caelles and K.-K. Maninis and L. {Van Gool}},
  title = {Iterative Deep Learning for Road Topology Extraction},
  journal = {British Machine Vision Conference (BMVC)},
  year = {2018}
}

The 2017 DAVIS Challenge on Video Object Segmentation

J. Pont-Tuset , F. Perazzi , S. Caelles , P. Arbeláez , A. Sorkine-Hornung , L. Van Gool

arXiv:1704.00675, 2017

[PDF] [Website] [BibTex]

@article{Pont-Tuset_arXiv_2017,
  author = {J. Pont-Tuset and F. Perazzi and S. Caelles and P. Arbel\'aez and A. Sorkine-Hornung and L. {Van Gool}},
  title = {The 2017 DAVIS Challenge on Video Object Segmentation},
  journal = {arXiv:1704.00675},
  year = {2017}
}

One-Shot Video Object Segmentation

S. Caelles* , K.-K. Maninis* , J. Pont-Tuset , L. Leal-Taixé , D. Cremers , L. Van Gool

Computer Vision and Pattern Recognition (CVPR), 2017

[PDF] [Website] [Code] [BibTex]

@inproceedings{Cae+17,
  Author  = {S. Caelles and K.-K. Maninis and J. Pont-Tuset and L. Leal-Taix\'e and D. Cremers and L. {Van Gool}},
  Title   = {One-Shot Video Object Segmentation},
  Booktitle = {Computer Vision and Pattern Recognition (CVPR)},
  Year    = {2017}
}

Pineapple

App that allows placing virtual food on your yet empty plate in a restaurant — so that you can always see what you get before you order it!

With N. Kobyshev.

HackZurich 2017

[Video] [More info] [Media] [Media]

Well2Go

Project carried out in collaboration with ESADE and IED students to improve public health by providing safe access to water. We developed a sensor network able to detect the state of the wells in developing countries, bringing real time information to a platform to ensure their maintenance. The project was the winner of the CBI Mediterranean 2015.

With A. Gran, F. Haddad, T. Mendes, B. Rosàs, N. Tsvetkov.

Challenge Based Innovation, IdeaSquare CERN 2016

[Video] [More info] [Media] [Media]

CityFlow

Platform that provides real-time information about hot areas in Barcelona, based on the information given by the geolocalization of Instagram posts uploaded in the city. It was selected for the Global Urban DataFest Contest.

With R. Boada, M. Dalmases, A. Fernàndez, A. Gómez, M. Martí.

Barcelona Smart Cities Hackathon 2015

[Video] [More info]

About

Latest News

Publications

Fun projects