Descriptive Attributes for Language-Based Object Keypoint Detection

Jerod Weinman; Serge Belongie; Stella Frank

doi:10.1007/978-3-031-44137-0_37

Descriptive Attributes for Language-Based Object Keypoint Detection

Jerod Weinman^*, Serge Belongie, Stella Frank

^*Corresponding author af dette arbejde

Pioneer AI (P1AI)

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Multimodal vision and language (VL) models have recently shown strong performance in phrase grounding and object detection for both zero-shot and finetuned cases. We adapt a VL model (GLIP) for keypoint detection and evaluate on NABirds keypoints. Our language-based keypoints-as-objects detector GLIP-KP outperforms baseline top-down keypoint detection models based on heatmaps and allows for zero- and few-shot evaluation. When fully trained, enhancing the keypoint names with descriptive attributes gives a significant performance boost, raising AP by as much as 6.0, compared to models without attribute information. Our model exceeds heatmap-based HRNet’s AP by 4.4 overall and 8.4 on keypoints with attributes. With limited data, attributes raise zero-/one-/few-shot test AP by 1.0/3.4/1.6, respectively, on keypoints with attributes.

Originalsprog	Engelsk
Titel	Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings
Redaktører	Henrik I. Christensen, Peter Corke, Renaud Detry, Jean-Baptiste Weibel, Markus Vincze
Forlag	Springer
Publikationsdato	2023
Sider	444-458
ISBN (Trykt)	9783031441363
DOI	https://doi.org/10.1007/978-3-031-44137-0_37
Status	Udgivet - 2023
Begivenhed	14th International Conference on Computer Vision Systems, ICVS 2023 - VIenna, Østrig Varighed: 27 sep. 2023 → 29 sep. 2023

Konference

Konference	14th International Conference on Computer Vision Systems, ICVS 2023
Land/Område	Østrig
By	VIenna
Periode	27/09/2023 → 29/09/2023

Navn	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vol/bind	14253 LNCS
ISSN	0302-9743

Bibliografisk note

Funding Information:
We thank Grant Van Horn for the mapping between NABirds and CUB, Jonathan M. Wells for helpful conversation, Vésteinn Snæbjarnarson for experimental assistance, and the reviewers for important feedback. This work was supported in part by the Pioneer Centre for AI, DNRF grant number P1.

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

Adgang til dokumentet

10.1007/978-3-031-44137-0_37

Citationsformater

Weinman, J., Belongie, S., & Frank, S. (2023). Descriptive Attributes for Language-Based Object Keypoint Detection. I H. I. Christensen, P. Corke, R. Detry, J-B. Weibel, & M. Vincze (red.), Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings (s. 444-458). Springer. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Bind 14253 LNCS https://doi.org/10.1007/978-3-031-44137-0_37

Descriptive Attributes for Language-Based Object Keypoint Detection. / Weinman, Jerod; Belongie, Serge ; Frank, Stella.

Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings. red. / Henrik I. Christensen; Peter Corke; Renaud Detry; Jean-Baptiste Weibel; Markus Vincze. Springer, 2023. s. 444-458 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bind 14253 LNCS).

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Weinman, J, Belongie, S & Frank, S 2023, Descriptive Attributes for Language-Based Object Keypoint Detection. i HI Christensen, P Corke, R Detry, J-B Weibel & M Vincze (red), Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings. Springer, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), bind 14253 LNCS, s. 444-458, 14th International Conference on Computer Vision Systems, ICVS 2023, VIenna, Østrig, 27/09/2023. https://doi.org/10.1007/978-3-031-44137-0_37

Weinman J, Belongie S , Frank S. Descriptive Attributes for Language-Based Object Keypoint Detection. I Christensen HI, Corke P, Detry R, Weibel J-B, Vincze M, red., Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings. Springer. 2023. s. 444-458. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bind 14253 LNCS). https://doi.org/10.1007/978-3-031-44137-0_37

Weinman, Jerod ; Belongie, Serge ; Frank, Stella. / Descriptive Attributes for Language-Based Object Keypoint Detection. Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings. red. / Henrik I. Christensen ; Peter Corke ; Renaud Detry ; Jean-Baptiste Weibel ; Markus Vincze. Springer, 2023. s. 444-458 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bind 14253 LNCS).

@inproceedings{7d94e5c6ff6a465a800c82107058a88f,

title = "Descriptive Attributes for Language-Based Object Keypoint Detection",

abstract = "Multimodal vision and language (VL) models have recently shown strong performance in phrase grounding and object detection for both zero-shot and finetuned cases. We adapt a VL model (GLIP) for keypoint detection and evaluate on NABirds keypoints. Our language-based keypoints-as-objects detector GLIP-KP outperforms baseline top-down keypoint detection models based on heatmaps and allows for zero- and few-shot evaluation. When fully trained, enhancing the keypoint names with descriptive attributes gives a significant performance boost, raising AP by as much as 6.0, compared to models without attribute information. Our model exceeds heatmap-based HRNet{\textquoteright}s AP by 4.4 overall and 8.4 on keypoints with attributes. With limited data, attributes raise zero-/one-/few-shot test AP by 1.0/3.4/1.6, respectively, on keypoints with attributes.",

keywords = "Attributes, Keypoint detection, Vision & language models",

author = "Jerod Weinman and Serge Belongie and Stella Frank",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.; 14th International Conference on Computer Vision Systems, ICVS 2023 ; Conference date: 27-09-2023 Through 29-09-2023",

year = "2023",

doi = "10.1007/978-3-031-44137-0_37",

language = "English",

isbn = "9783031441363",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "444--458",

editor = "Christensen, {Henrik I.} and Peter Corke and Renaud Detry and Jean-Baptiste Weibel and Markus Vincze",

booktitle = "Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings",

address = "Switzerland",

}

TY - GEN

T1 - Descriptive Attributes for Language-Based Object Keypoint Detection

AU - Weinman, Jerod

AU - Belongie, Serge

AU - Frank, Stella

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

PY - 2023

Y1 - 2023

N2 - Multimodal vision and language (VL) models have recently shown strong performance in phrase grounding and object detection for both zero-shot and finetuned cases. We adapt a VL model (GLIP) for keypoint detection and evaluate on NABirds keypoints. Our language-based keypoints-as-objects detector GLIP-KP outperforms baseline top-down keypoint detection models based on heatmaps and allows for zero- and few-shot evaluation. When fully trained, enhancing the keypoint names with descriptive attributes gives a significant performance boost, raising AP by as much as 6.0, compared to models without attribute information. Our model exceeds heatmap-based HRNet’s AP by 4.4 overall and 8.4 on keypoints with attributes. With limited data, attributes raise zero-/one-/few-shot test AP by 1.0/3.4/1.6, respectively, on keypoints with attributes.

AB - Multimodal vision and language (VL) models have recently shown strong performance in phrase grounding and object detection for both zero-shot and finetuned cases. We adapt a VL model (GLIP) for keypoint detection and evaluate on NABirds keypoints. Our language-based keypoints-as-objects detector GLIP-KP outperforms baseline top-down keypoint detection models based on heatmaps and allows for zero- and few-shot evaluation. When fully trained, enhancing the keypoint names with descriptive attributes gives a significant performance boost, raising AP by as much as 6.0, compared to models without attribute information. Our model exceeds heatmap-based HRNet’s AP by 4.4 overall and 8.4 on keypoints with attributes. With limited data, attributes raise zero-/one-/few-shot test AP by 1.0/3.4/1.6, respectively, on keypoints with attributes.

KW - Attributes

KW - Keypoint detection

KW - Vision & language models

UR - http://www.scopus.com/inward/record.url?scp=85174519994&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-44137-0_37

DO - 10.1007/978-3-031-44137-0_37

M3 - Article in proceedings

AN - SCOPUS:85174519994

SN - 9783031441363

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 444

EP - 458

BT - Computer Vision Systems - 14th International Conference, ICVS 2023, Proceedings

A2 - Christensen, Henrik I.

A2 - Corke, Peter

A2 - Detry, Renaud

A2 - Weibel, Jean-Baptiste

A2 - Vincze, Markus

PB - Springer

T2 - 14th International Conference on Computer Vision Systems, ICVS 2023

Y2 - 27 September 2023 through 29 September 2023

ER -