Nataniel Ruiz

I am a Research Scientist at Google. I completed my PhD at Boston University, advised by Professor and Dean of the College of Arts and Sciences Stan Sclaroff. I did my Masters at Georgia Tech with James Rehg and my Bachelors at Ecole Polytechnique in Paris. Prior to joining Google I interned at Apple, Amazon and NEC Labs. My primary research focus is computer vision and machine learning.

CV | Google Scholar | GitHub | Twitter | LinkedIn
natanielruiz [at] google [dot] com

Currently, my main interests include generative models, diffusion models, personalization of generative models, simulation and beneficial adversarial attacks.

Magic Insert: Style-Aware Drag-and-Drop
Nataniel Ruiz, Yuanzhen Li, Neal Wadhwa, Yael Pritch, Michael Rubinstein, David E. Jacobs, Shlomi Fruchter
arXiv preprint arXiv:2407.02489, 2024
website | demo | dataset | tweet

RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, Wen-Sheng Chu
arXiv preprint arXiv:2405.17401, 2024
website | demo | code | tweet

ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, Varun Jampani
European Conference on Computer Vision (ECCV), 2024
website | tweet | unofficial code

RealFill: Reference-driven Generation for Authentic Image Completion
Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E. Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, Michael Rubinstein
SIGGRAPH (Journal Track), 2024
website | two minute papers | data | tweet | code

HyperDreamBooth: Hypernetworks for Fast Personalization of Text-to-Image Models
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, Kfir Aberman
Conference on Computer Vision and Pattern Recognition (CVPR), 2024
website | tweet

Platypus: Quick, Cheap, and Powerful Refinement of LLMs
Ariel N. Lee, Cole J. Hunter, Nataniel Ruiz
Neural Information Processing Systems Workshop (NeurIPS Workshop​)​, 2023
website | tweet | models | dataset | code

SuTI: Subject-driven Text-to-Image Generation via Apprenticeship Learning
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen
Conference on Neural Information Processing Systems (NeurIPS), 2023
website | google cloud launch

StyleDrop: Text-to-Image Generation in Any Style
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, Dilip Krishnan
Conference on Neural Information Processing Systems (NeurIPS), 2023
website | tweet | google cloud launch | unofficial code

DreamBooth3D: Subject-driven Text-to-3D Generation
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, Varun Jampani
International Conference on Computer Vision (ICCV), 2023
website | demo video | tweet

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman
Conference on Computer Vision and Pattern Recognition (CVPR), 2023 (Best Student Paper Honorable Mention Award - 0.25% award rate)
website | wikipedia | corridor crew video | tweet | diffusers code | alt unofficial code | google cloud launch

Simulating to Learn: Using Adaptive Simulation to Train, Test and Understand Neural Networks
Nataniel Ruiz
Boston University, 2023 (PhD Thesis)
This thesis presents new insights into the use of adaptive simulation to train and test machine learning models, addressing the key obstacle of collecting annotated and high-quality real-world data. It presents five novel methods for adapting simulated data distributions to improve the training and testing of neural networks.

Practical Disruption of Image Translation Deepfake Networks
Nataniel Ruiz, Sarah A. Bargal, Cihang Xie, Stan Sclaroff
AAAI Conference on Artificial Intelligence, 2023

Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
Ariel N. Lee, Sarah A. Bargal, Janavi Kasera, Stan Sclaroff, Kate Saenko, Nataniel Ruiz
arXiv preprint arXiv:2306.17848, 2023
website | Superimposed Masked Dataset | Realistic Occlusion Dataset | tweet

Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing
Nataniel Ruiz, Sarah A. Bargal, Cihang Xie, Kate Saenko, Stan Sclaroff
Conference on Neural Information Processing Systems (NeurIPS), 2022
website | Dataset

Human Body Measurement Estimation with Adversarial Augmentation
Nataniel Ruiz, Miriam Bellver, Timo Bolkart, Ambuj Arora, Ming C. Lin, Javier Romero, Raja Bala
International Conference on 3D Vision (3DV), 2022
website | poster | video | dataset

Simulated Adversarial Testing of Face Recognition Models
Nataniel Ruiz, Adam Kortylewski, Weichao Qiu, Cihang Xie, Sarah A. Bargal, Alan Yuille, Stan Sclaroff
Conference on Computer Vision and Pattern Recognition (CVPR), 2022

MorphGAN: One-Shot Face Synthesis GAN for Detecting Recognition Bias
Nataniel Ruiz, Barry-John Theobald, Anurag Ranjan, Ahmed H. Abdelaziz, Nicholas Apostoloff
British Machine Vision Conference (BMVC), 2021

Protecting Against Image Translation Deepfakes by Leaking Universal Perturbations from Black-Box Neural Networks
Nataniel Ruiz, Sarah A. Bargal, Stan Sclaroff
arxiv, 2021

Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System
Nataniel Ruiz, Hao Yu, Danielle A. Allessio, Mona Jalal, Thomas Murray, John J. Magee, Jacob R. Whitehill, Vitaly Ablavsky, Ivon Arroyo, Beverly P. Woolf, Stan Sclaroff, Margrit Betke
IEEE International Conference on Automatic Face and Gesture Recognition (FG), 2021
IEEE Transactions on Biometrics, Identity and Behavior (T-BIOM), 2022
journal paper
(Oral and Best Poster Award - 4% award rate)

Disrupting DeepFakes: Adversarial Attacks Against Conditional Image Translation Networks and Facial Manipulation Systems
Nataniel Ruiz, Sarah A. Bargal, Stan Sclaroff
CVPR Workshop on Adversarial Machine Learning in Computer Vision and ECCV Workshop on Advances in Image Manipulation, 2020
podcast | code | video demo

Detecting Attended Visual Targets in Video
Eunji Chong, Yongxin Wang, Nataniel Ruiz, James M. Rehg
Conference on Computer Vision and Pattern Recognition (CVPR), 2020
code

Learning To Simulate
Nataniel Ruiz, Samuel Schulter, Manmohan Chandraker
International Conference on Learning Representations (ICLR), 2019
poster

Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency
Eunji Chong, Nataniel Ruiz, Yongxin Wang, Yun Zhang, Agata Rozga, James M. Rehg
European Conference on Computer Vision (ECCV), 2018
poster | bibtex

Fine-Grained Head Pose Estimation Without Keypoints
Nataniel Ruiz, Eunji Chong, James M. Rehg
Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2018 (Oral)
code | video demo | poster | bibtex

Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
Meera Hahn, Nataniel Ruiz, Jean-Baptiste Alayrac, Ivan Laptev, James M. Rehg
arXiv, 2018

Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment
Eunji Chong, Katha Chanda, Zhefan Ye, Audrey Southerland, Nataniel Ruiz, Rebecca M. Jones, Agata Rozga, James M. Rehg
UbiComp and Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2017
(Oral Presentation and Distinguished Paper Award - 3% award rate)
bibtex

Dockerface: an Easy to Install and Use Faster R-CNN Face Detector in a Docker Container
Nataniel Ruiz, James M. Rehg
arXiv, 2017
code | bibtex

android-yolo
Nataniel Ruiz
video demo | app apk
Real-time object detection on Android using the YOLO network with TensorFlow.